A method, system, terminal and storage medium for classifying sliced images

Abstract

The present invention discloses a method, system, terminal and storage medium for classifying sliced images. The method includes: preprocessing a sliced image to be processed to obtain a target sliced image; constructing a sliced image classification model, training the sliced image classification model to obtain a target model, inputting the target sliced image into an instance feature extraction module of the target model for feature extraction to obtain first instance features, inputting the first instance features into an instance feature learning module of the target model for feature recombination and learning to obtain second instance features; inputting the second instance features into an instance feature aggregation module of the target model for dimension elevation and aggregation to obtain packet-level features, and inputting the packet-level features into a packet prediction module of the target model for classification to obtain a classification result of the sliced image to be processed. The present invention applies multiple scanning methods for feature complementarity and effectively highlights key instances through global context attention, realizing accurate classification of sliced images.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular, to a method, system, terminal, and computer-readable storage medium for classifying sliced images. Background Art

[0002] Digital slices are generated by using a fully automatic microscope scanning system in combination with a virtual slicing software system to scan and seamlessly stitch traditional glass slices to generate a whole-field digital slice image (Whole Slide Image, WSI). In the practice of pathology medical treatment, teaching, and research, digital slices have all the functions of traditional slices and have the advantages of being unrestricted by space and time. A digital slice is not a static picture. It contains all the lesion information on the glass slice. On a computer, just like under a microscope, it can be observed at different magnifications and, within a certain range, achieve stepless continuous zooming to view the slice.

[0003] Due to the high-resolution characteristics of the whole-field digital slice image, it is difficult to obtain accurate results by directly analyzing the entire WSI. Therefore, many researchers have started to use multi-instance learning to analyze WSI to achieve tasks such as classification, grading, and survival prediction. However, the proportion of target regions in WSI is relatively small, resulting in most attention mechanism-based multi-instance learning methods being unable to accurately analyze WSI.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method, system, terminal, and computer-readable storage medium for classifying sliced images, aiming to solve the problem that most attention mechanism-based multi-instance learning methods in the prior art cannot accurately classify whole-field digital slice images.

[0006] To achieve the above object, the present invention provides a method for classifying sliced images, and the method for classifying sliced images includes the following steps:

[0007] Obtain a sliced image to be processed, and perform preprocessing on the sliced image to be processed to obtain a target sliced image;

[0008] Construct a sliced image classification model, train and test the sliced image classification model to obtain a target model, and the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module, and a bag prediction module;

[0009] Input the target slice image into the instance feature extraction module of the target model for feature extraction to obtain the first instance feature, and input the first instance feature into the instance feature learning module of the target model for feature recombination and learning to obtain the second instance feature;

[0010] Input the second instance feature into the instance feature aggregation module of the target model for dimension elevation and aggregation to obtain the packet-level feature, and input the packet-level feature into the packet prediction module of the target model for classification to obtain the classification result of the slice image to be processed.

[0011] Optionally, for the slice image classification method, wherein the step of inputting the target slice image into the instance feature extraction module of the target model for feature extraction to obtain the first instance feature specifically includes:

[0012] Input the target slice image into the instance feature extraction module for cropping to obtain a plurality of non-overlapping instances, and remove the instances that do not contain tissue background among the plurality of instances by a threshold method to obtain target instances;

[0013] Input the target instances into a pre-trained ResNet50 model for feature extraction to obtain initial features, and input the initial features into a multi-layer perceptron for dimension reduction to obtain the first instance feature.

[0014] Optionally, for the slice image classification method, wherein the instance feature learning module includes: a feature recombination sub-module and an MSM sub-module;

[0015] The step of inputting the first instance feature into the instance feature learning module of the target model for feature recombination and learning to obtain the second instance feature specifically includes:

[0016] Input the first instance feature into three branches of the feature recombination sub-module for recombination to generate three new feature sequences;

[0017] Input the three new feature sequences into the MSM sub-module for feature learning, fuse the learned features to obtain key instance features, and input the key instance features into a linear layer for linear transformation, and obtain the second instance feature based on the transformed features and the first instance feature.

[0018] Optionally, for the slice image classification method, wherein the step of inputting the first instance feature into three branches of the feature recombination module for recombination to generate three new feature sequences specifically includes:

[0019] Input the first instance feature into the first branch of the feature recombination sub-module for original scanning to obtain a first new feature sequence that retains the original sequence structure;

[0020] Input the first instance feature into the second branch of the feature recombination sub-module for grid scanning, reshape to obtain four grid subsequences in different directions, reorder the four grid subsequences to obtain a second new feature sequence after recombination;

[0021] Input the first instance feature into the third branch of the feature recombination sub-module for hierarchical scanning, reshape to obtain two hierarchical subsequences in different directions, reorder the two hierarchical subsequences to obtain a third new feature sequence after recombination.

[0022] Optionally, in the slice image classification method, the MSM sub-module includes three branches, and the three branches of the MSM sub-module correspond one-to-one with the three branches of the feature recombination module;

[0023] Wherein, each branch of the MSM sub-module includes a Mamba block and a GCA block, the Mamba block includes a linear layer, a convolutional layer, a SiLU activation function, and a state space model, and the GCA block includes layer normalization and an MLP layer.

[0024] Optionally, in the slice image classification method, input the three new feature sequences into the MSM sub-module for feature learning, fuse the learned features to obtain a key instance feature, and input the key instance feature into a linear layer for linear transformation, and obtain a second instance feature based on the transformed feature and the first instance feature, specifically including:

[0025] Input the three new feature sequences into the three branches of the MSM sub-module respectively, perform feature learning through the Mamba block and GCA block of each branch to obtain the learned features of each branch;

[0026] Fuse the learned features of each branch to obtain a key instance feature, input the key instance feature into a linear layer for linear transformation to obtain a transformed feature, and perform matrix addition on the transformed feature and the first instance feature to obtain a second instance feature.

[0027] Optionally, in the slice image classification method, input the second instance feature into the instance feature aggregation module of the target model for dimension elevation and aggregation to obtain a packet-level feature, specifically including:

[0028] Input the second instance feature into the linear layer of the instance feature aggregation module of the target model for dimensionality increase to obtain the feature after dimensionality increase;

[0029] Use the gated attention mechanism to aggregate the feature after dimensionality increase to obtain the aggregated packet-level feature.

[0030] In addition, to achieve the above object, the present invention further provides a sliced image classification system, wherein the sliced image classification system includes:

[0031] A sliced image acquisition module, configured to acquire a sliced image to be processed, perform preprocessing on the sliced image to be processed to obtain a target sliced image;

[0032] A classification model construction module, configured to construct a sliced image classification model, train and test the sliced image classification model to obtain a target model, and the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module, and a packet prediction module;

[0033] A feature extraction and learning module, configured to input the target sliced image into the instance feature extraction module of the target model for feature extraction to obtain a first instance feature, and input the first instance feature into the instance feature learning module of the target model for recombination and learning of the feature to obtain a second instance feature;

[0034] A feature aggregation and classification module, configured to input the second instance feature into the instance feature aggregation module of the target model for dimensionality increase and aggregation to obtain a packet-level feature, and input the packet-level feature into the packet prediction module of the target model for classification to obtain the classification result of the sliced image to be processed.

[0035] In addition, to achieve the above object, the present invention further provides a terminal, wherein the terminal includes: a memory, a processor, and a sliced image classification program stored on the memory and executable on the processor, and when the sliced image classification program is executed by the processor, the steps of the sliced image classification method as described above are implemented.

[0036] In addition, to achieve the above object, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a sliced image classification program, and when the sliced image classification program is executed by a processor, the steps of the sliced image classification method as described above are implemented.

[0037] In the present invention, the slice image to be processed is preprocessed to obtain a target slice image; a slice image classification model is constructed, and the slice image classification model is trained to obtain a target model. The target slice image is input into the instance feature extraction module of the target model for feature extraction to obtain a first instance feature. The first instance feature is input into the instance feature learning module of the target model for feature recombination and learning to obtain a second instance feature; the second instance feature is input into the instance feature aggregation module of the target model for dimension elevation and aggregation to obtain a packet-level feature, and the packet-level feature is input into the packet prediction module of the target model for classification to obtain the classification result of the slice image to be processed. The present invention applies multiple scanning methods for feature complementarity and effectively highlights key instances through global context attention, realizing accurate classification of slice images. Description of the Drawings

[0038] Figure 1 is a flowchart of a preferred embodiment of the slice image classification method of the present invention;

[0039] Figure 2 is an overall architecture diagram of the target model in the slice image classification method of the present invention;

[0040] Figure 3 is a schematic diagram of the multi-directional scanning mechanism in the slice image classification method of the present invention;

[0041] Figure 4 is an architecture diagram of the GCA block in the slice image classification method of the present invention;

[0042] Figure 5 is a structural diagram of a preferred embodiment of the slice image classification system of the present invention;

[0043] Figure 6 is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Embodiments

[0044] The present application provides a slice image classification method, system and terminal. To make the purpose, technical solution and effect of the present application clearer and more definite, the following further describes the present application in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] Those skilled in the art of the present technology can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the technical field to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted in an idealized or overly formal sense unless specifically defined as here.

[0046] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, such descriptions of "first", "second", etc. are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions between various embodiments can be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions results in contradictions or inability to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0047] The slice image classification method according to a preferred embodiment of the present invention, as Figure 1 and Figure 2 shown, the slice image classification method includes the following steps:

[0048] Step S10: Obtain a slice image to be processed, preprocess the slice image to be processed, and obtain a target slice image.

[0049] Specifically, obtain a slice image to be processed (classified), preprocess the slice image to be processed, and the preprocessing process may include random cropping, data cleaning, data standardization, etc., and finally obtain a target slice image for inputting into the model.

[0050] Step S20: Construct a slice image classification model, train and test the slice image classification model, and obtain a target model, where the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module, and a packet prediction module.

[0051] Specifically, construct a slice image classification model as Figure 2 shown, and train and test the slice image classification model to obtain a target model.

[0052] Before training and testing the slice image classification model, it further includes:

[0053] Obtain historical slice images and the corresponding historical classification results of the historical slice images, use the historical slice images as training samples, use the historical classification results as labels, and construct a data set;

[0054] Divide the data set into a training set, a test set, and a validation set according to a preset ratio. The training set is used to train the slice image classification model, the validation set is used to evaluate the slice image classification model in each round of training, and the test set is used to evaluate the trained slice image classification model.

[0055] In an embodiment, the historical slice images for training and testing and the corresponding historical classification results of the historical slice images are from datasets of two different cancers, namely: 1. The Camelyon16 dataset for breast cancer lymph node metastasis detection, which contains 240 WSIs from patients with breast cancer lymph node metastasis and 159 WSIs from normal patients. 2. The TCGA-Lung dataset for cancer subtype classification, which consists of two subtypes, lung squamous cell carcinoma (LUSC) and lung adenocarcinoma (LUAD), with the number of WSIs for each subtype being 527 and 511 respectively. These two datasets are from public datasets and can be freely downloaded and used.

[0056] Further, in the Camelyon16 dataset, the given training set is randomly divided into a training set and a validation set at a ratio of 4:1, and the test set is used for testing. For the TCGA-Lung dataset, it is randomly divided into a training set, a validation set, and a test set at a ratio of 3:1:1 for experiments. Specifically, the model is trained using the PyTorch framework on a TITAN RTX 4090 GPU with 24GB of memory. During the training process, the initial learning rate is set to and a learning rate decay mechanism with the decay coefficient is utilized. In addition, to optimize the performance of the model, this application uses the Lookahead (Lookahead is an optimization algorithm that improves the training process of deep learning models by maintaining two sets of weights instead of one) optimizer to optimize the model, and uses the cross-entropy loss as the objective function when training the model. At the same time, 200 batches are set for the experiment, and in each batch, the batch size is set to 1 to accommodate the phenomenon that the number of instances cropped from each WSI is different.

[0057] It can be understood that this application further uses accuracy, sensitivity, precision, F1 score (F1), area under the curve (AUC), and Kappa score (Kappa) as the classification performance evaluation criteria. These metrics can evaluate the performance of the slice image classification model from various aspects, and finally obtain the trained target model.

[0058] Step S30: Input the target slice image into the instance feature extraction module of the target model for feature extraction to obtain the first instance feature, and input the first instance feature into the instance feature learning module of the target model for feature recombination and learning to obtain the second instance feature.

[0059] Inputting the target slice image into the instance feature extraction module of the target model for feature extraction to obtain the first instance feature specifically includes:

[0060] Inputting the target slice image into the instance feature extraction module for cropping to obtain a plurality of non-overlapping instances, and removing the instances that do not contain tissue background among the plurality of instances through a threshold method to obtain target instances;

[0061] Inputting the target instances into a pre-trained ResNet50 model for feature extraction to obtain initial features, and inputting the initial features into a multi-layer perceptron for dimensionality reduction to obtain the first instance feature.

[0062] In this embodiment, given a target slice image, the first step is to crop it into non-overlapping image patches, called instances, and use a threshold method to remove the instances that do not contain tissue background. The size of each instance is 256×256. Then, the pre-trained ResNet50 model on ImageNet is used to extract features from each instance. The dimension of the feature vector of each instance is 1024. To reduce feature redundancy and computational overhead, a multi-layer perceptron is used to reduce the feature dimension from 1024 to 256 to obtain the first instance feature.

[0063] Further, the instance feature learning module includes: a feature recombination sub-module and an MSM sub-module; the process of inputting the first instance feature into the instance feature learning module of the target model for feature recombination and learning to obtain the second instance feature specifically includes:

[0064] Inputting the first instance feature into three branches of the feature recombination sub-module for recombination to generate three new feature sequences; inputting the three new feature sequences into the MSM sub-module for feature learning, fusing the learned features to obtain key instance features, and inputting the key instance features into a linear layer for linear transformation, and obtaining the second instance feature based on the transformed features and the first instance feature.

[0065] Further, the process of inputting the first instance feature into three branches of the feature recombination module for recombination to generate three new feature sequences specifically includes:

[0066] Inputting the first instance feature into the first branch of the feature recombination sub-module for original scanning to obtain a first new feature sequence that retains the original sequence structure;

[0067] Inputting the first instance feature into the second branch of the feature recombination submodule for grid scanning, reshaping to obtain four grid subsequences in different directions, and reordering the four grid subsequences to obtain a reorganized second new feature sequence;

[0068] The first instance feature is input into the third branch of the feature recombination submodule for hierarchical scanning, and two hierarchical subsequences in different directions are reshaped. The two hierarchical subsequences are reordered to obtain a recombined third new feature sequence.

[0069] like Figure 3 As shown, it can be understood that the present application reorganizes the extracted features to regenerate a new sequence. During the reorganization process, if the instance length is not enough to divide the scanning direction, the sequence is padded with 0. In particular, the present application retains the original sequence structure in the first branch to capture global feature information, which can make up for the feature loss of the overall sequence caused by the sequence reshaping in the other two branches. In the second branch, a grid scanning strategy is adopted to first divide the sequence into four subsequences in different directions, and then reorder the four subsequences into a new sequence. Since the input features are reorganized, after learning the feature information from the Mamba block, the output features are rearranged to be consistent with the feature order of the first branch. Similarly, in the third branch, a hierarchical scanning strategy is first adopted to reshape the instance features into two subsequences in different directions, and then reorder them into a new sequence. Finally, the output features of the third branch are reordered like the second branch. At the same time, the third branch can also supplement the sequence features of the second branch to a certain extent. Therefore, the three branches of the entire MSM module can better mine the feature associations between sequences.

[0070] Furthermore, the MSM (Multi-scan Mamba) module consists of three branches, each containing a Mamba block and a GCA block. The Mamba block is a deep learning architecture based on the Structured Space Model (SSM) designed to efficiently process long sequence data and is used to effectively model the dependencies between instances, while the GCA (global context attention) block is used to strengthen the relationship between instances and highlight the importance of key instances.

[0071] Specifically, the MSM sub-module includes three branches, where the three branches of the MSM sub-module correspond one-to-one with the three branches of the feature recombination module; among them, each branch of the MSM sub-module includes a Mamba block and a GCA block, the Mamba block includes a linear layer, a convolutional layer, a SiLU activation function, and a state space model, and the GCA block includes layer normalization and an MLP layer.

[0072] Furthermore, input the three new feature sequences into the MSM sub-module for feature learning, fuse the learned features to obtain key instance features, and input the key instance features into a linear layer for linear transformation, and obtain second instance features based on the transformed features and the first instance features, specifically including:

[0073] Input the three new feature sequences into the three branches of the MSM sub-module respectively, and perform feature learning through the Mamba block and the GCA block of each branch to obtain the learned features of each branch;

[0074] Fuse the learned features of each branch to obtain key instance features, input the key instance features into a linear layer for linear transformation to obtain transformed features, and perform matrix addition on the transformed features and the first instance features to obtain second instance features.

[0075] The learning process of the MSM sub-module is as follows: Input the new feature sequence into the first branch of the MSM sub-module to perform feature learning through the linear layer and the convolutional layer, input the extracted features into the state space model after passing through the SiLU activation function, and output the first Mamba feature. Then, input the new feature sequence into the second branch of the MSM sub-module to perform transformation through the linear layer and the SiLU activation function, and output the second Mamba feature. Finally, perform matrix multiplication on the first learned feature and the second learned feature to obtain the target Mamba feature of the corresponding branch in the MSM sub-module.

[0076] As Figure 4 shown, the learning process of the GCA block is as follows: Perform feature fusion on the target Mamba feature learned from the Mamba branch and input it into the GCA block. At this time, first use the layer normalization layer to normalize the input features, and use its mean value as the weight. After passing through the MLP layer, perform matrix multiplication on the output and the original input features to highlight the key instances. Then use a linear layer to obtain the final output feature of the GCA block (i.e., the transformed feature). Finally, perform matrix addition on the transformed feature and the first instance feature to obtain the second instance feature.

[0077] Step S40: Input the second instance feature into the instance feature aggregation module of the target model for dimension elevation and aggregation to obtain a packet-level feature, and input the packet-level feature into the packet prediction module of the target model for classification to obtain the classification result of the to-be-processed slice image.

[0078] It should be noted that feature aggregation is a key step in the multi-instance learning method because it enables the model to fully capture the relationships between and within instances. This process is crucial for effectively selecting discriminative instances and accurately predicting the category of the WSI. In the method proposed in this application, multiple MSM modules are used to achieve feature aggregation. Therefore, multiple second instance features are obtained here.

[0079] Further, inputting the second instance feature into the instance feature aggregation module of the target model for dimension elevation and aggregation to obtain a packet-level feature specifically includes:

[0080] Input the second instance feature into the linear layer of the instance feature aggregation module of the target model for dimension elevation to obtain a dimension-elevated feature;

[0081] Use a gated attention mechanism to perform feature aggregation on the dimension-elevated feature to obtain an aggregated packet-level feature.

[0082] In this embodiment, after feature learning by the MSM module, the learned second instance feature is first dimension-elevated using a linear layer to enable the feature information to be fully expressed. Then, the dimension-elevated instance feature is fed into the feature aggregation module for feature aggregation. In this process, a general gated attention mechanism is used to achieve feature aggregation to form a packet-level feature. This method can further make the packet-level feature more inclined to the features of key instances. After forming the packet-level feature, a classifier is used to classify the packet-level feature to achieve the final WSI classification.

[0083] It can be seen that the present invention mainly includes two parts: (1) Mamba module: A multi-directional scanning method is proposed to improve the problem that the original Mamba scanning method cannot effectively mine the spatial information in the sequence. To supplement feature details, multiple scanning methods are applied for feature complementarity to achieve full mining of features and amplify the importance of key instances. (2) Global context attention module: A GCA module is designed, which can effectively highlight key instances, enabling the network to better learn the target regions in the WSI and achieve accurate classification of the WSI. Experimental results on two public datasets, TCGA-Lung and Camelyon16, show that the proposed method is superior to the existing state-of-the-art methods and can better assist pathologists in classifying pathological slices.

[0084] Furthermore, as Figure 5 shown, based on the above slice image classification method, the present invention also correspondingly provides a slice image classification system, wherein the slice image classification system includes:

[0085] A slice image acquisition module 51, configured to acquire a slice image to be processed, preprocess the slice image to be processed, and obtain a target slice image;

[0086] A classification model construction module 52, configured to construct a slice image classification model, train and test the slice image classification model, and obtain a target model, where the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module, and a packet prediction module;

[0087] A feature extraction and learning module 53, configured to input the target slice image into the instance feature extraction module of the target model for feature extraction to obtain a first instance feature, and input the first instance feature into the instance feature learning module of the target model for recombination and learning of the features to obtain a second instance feature;

[0088] A feature aggregation and classification module 54, configured to input the second instance feature into the instance feature aggregation module of the target model for dimension elevation and aggregation to obtain a packet-level feature, and input the packet-level feature into the packet prediction module of the target model for classification to obtain a classification result of the slice image to be processed.

[0089] Furthermore, as Figure 6 shown, based on the above slice image classification method and system, the present invention also correspondingly provides a terminal, where the terminal includes a processor 10, a memory 20, and a display 30. Figure 6 Only some components of the terminal are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0090] The memory 20 may be an internal storage unit of the terminal in some embodiments, such as a hard disk or memory of the terminal. The memory 20 may also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the terminal. Further, the memory 20 may also include both the internal storage unit of the terminal and the external storage device. The memory 20 is used to store application software installed on the terminal and various types of data, such as program codes for installing the terminal, etc. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, a slice image classification program 40 is stored on the memory 20, and the slice image classification program 40 can be executed by the processor 10, so as to implement the slice image classification method in this application.

[0091] The processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chips in some embodiments, and is used to run program codes stored in the memory 20 or process data, such as executing the slice image classification method, etc.

[0092] The display 30 may be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) toucher, etc. in some embodiments. The display 30 is used to display information on the terminal and to display a visual user interface. Components of the terminal communicate with each other through a system bus.

[0093] In one embodiment, when the processor 10 executes the slice image classification program 40 in the memory 20, the following steps are implemented:

[0094] Obtain a slice image to be processed, preprocess the slice image to be processed to obtain a target slice image;

[0095] Construct a slice image classification model, train and test the slice image classification model to obtain a target model, and the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module, and a packet prediction module;

[0096] Input the target slice image into the instance feature extraction module of the target model for feature extraction to obtain a first instance feature, and input the first instance feature into the instance feature learning module of the target model for feature recombination and learning to obtain a second instance feature;

[0097] Input the second instance feature into the instance feature aggregation module of the target model for dimension elevation and aggregation to obtain a packet-level feature, and input the packet-level feature into the packet prediction module of the target model for classification to obtain the classification result of the slice image to be processed.

[0098] Among them, inputting the target slice image into the instance feature extraction module of the target model for feature extraction to obtain a first instance feature specifically includes:

[0099] Input the target slice image into the instance feature extraction module for cropping to obtain a plurality of non-overlapping instances, and remove the instances that do not contain tissue background among the plurality of instances through a threshold method to obtain target instances;

[0100] Input the target instance into a pre-trained ResNet50 model for feature extraction to obtain an initial feature, and input the initial feature into a multi-layer perceptron for dimension reduction to obtain a first instance feature.

[0101] Among them, the instance feature learning module includes: a feature recombination sub-module and an MSM sub-module;

[0102] Inputting the first instance feature into the instance feature learning module of the target model for feature recombination and learning to obtain a second instance feature specifically includes:

[0103] Input the first instance feature into three branches of the feature recombination sub-module for recombination to generate three new feature sequences;

[0104] Input the three new feature sequences into the MSM sub-module for feature learning, fuse the learned features to obtain a key instance feature, and input the key instance feature into a linear layer for linear transformation, and obtain a second instance feature based on the transformed feature and the first instance feature.

[0105] Among them, inputting the first instance feature into three branches of the feature recombination module for recombination to generate three new feature sequences specifically includes:

[0106] Input the first instance feature into the first branch of the feature recombination sub-module for original scanning to obtain a first new feature sequence that retains the original sequence structure;

[0107] Input the first instance feature into the second branch of the feature recombination sub-module for grid scanning, reshape to obtain four grid subsequences in different directions, and reorder the four grid subsequences to obtain a recombined second new feature sequence;

[0108] Input the first instance feature into the third branch of the feature recombination sub-module for hierarchical scanning, reshape it to obtain two hierarchical subsequences in different directions, reorder the two hierarchical subsequences, and obtain the recombined third new feature sequence.

[0109] Among them, the MSM sub-module includes three branches, and among them, the three branches of the MSM sub-module correspond one-to-one with the three branches of the feature recombination module;

[0110] Among them, each branch of the MSM sub-module includes a Mamba block and a GCA block. The Mamba block includes a linear layer, a convolutional layer, a SiLU activation function, and a state space model. The GCA block includes layer normalization and an MLP layer.

[0111] Among them, input the three new feature sequences into the MSM sub-module for feature learning, fuse the learned features to obtain the key instance feature, and input the key instance feature into a linear layer for linear transformation, and obtain the second instance feature based on the transformed feature and the first instance feature. Specifically, it includes:

[0112] Input the three new feature sequences into the three branches of the MSM sub-module respectively, and perform feature learning through the Mamba block and the GCA block of each branch to obtain the learned features of each branch;

[0113] Fuse the learned features of each branch to obtain the key instance feature, input the key instance feature into a linear layer for linear transformation to obtain the transformed feature, and perform matrix addition on the transformed feature and the first instance feature to obtain the second instance feature.

[0114] Among them, input the second instance feature into the instance feature aggregation module of the target model for dimension elevation and aggregation to obtain the packet-level feature. Specifically, it includes:

[0115] Input the second instance feature into the linear layer of the instance feature aggregation module of the target model for dimension elevation to obtain the dimension-elevated feature;

[0116] Use the gated attention mechanism to perform feature aggregation on the dimension-elevated feature to obtain the aggregated packet-level feature.

[0117] The present invention also provides a computer-readable storage medium. Among them, the computer-readable storage medium stores a sliced image classification program, and when the sliced image classification program is executed by a processor, it implements the steps of the sliced image classification method described above.

[0118] In summary, the present invention proposes a method, system and terminal for classifying slice images. The method includes: obtaining a slice image to be processed, preprocessing the slice image to be processed to obtain a target slice image; constructing a slice image classification model, training and testing the slice image classification model to obtain a target model, where the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module and a packet prediction module; inputting the target slice image into the instance feature extraction module of the target model for feature extraction to obtain a first instance feature, and inputting the first instance feature into the instance feature learning module of the target model for recombination and learning of features to obtain a second instance feature; inputting the second instance feature into the instance feature aggregation module of the target model for dimension elevation and aggregation to obtain a packet-level feature, and inputting the packet-level feature into the packet prediction module of the target model for classification to obtain a classification result of the slice image to be processed. The present invention applies multiple scanning methods for feature complementarity, and effectively highlights key instances through global context attention, realizing accurate classification of slice images.

[0119] It should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or terminal. Without more limitations, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or terminal including the element.

[0120] Of course, those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present invention can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0121] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or changes can be made according to the above description. All such improvements and changes should fall within the protection scope of the appended claims of the present invention.

Claims

1. A method for classifying sliced images, characterized in that, The described slice image classification method includes: Obtain the slice image to be processed, preprocess the slice image to be processed to obtain a target slice image; Construct a slice image classification model, train and test the slice image classification model to obtain a target model, and the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module, and a packet prediction module; Input the target slice image into the instance feature extraction module of the target model for feature extraction to obtain a first instance feature, and input the first instance feature into the instance feature learning module of the target model for feature recombination and learning to obtain a second instance feature; The step of inputting the target slice image into the instance feature extraction module of the target model for feature extraction to obtain a first instance feature specifically includes: Input the target slice image into the instance feature extraction module for cropping to obtain a plurality of non-overlapping instances, and remove the instances that do not contain tissue background among the plurality of instances by a threshold method to obtain target instances; Input the target instances into a pre-trained ResNet50 model for feature extraction to obtain initial features, and input the initial features into a multi-layer perceptron for dimensionality reduction to obtain a first instance feature; The instance feature learning module includes: a feature recombination sub-module and an MSM sub-module; The step of inputting the first instance feature into the instance feature learning module of the target model for feature recombination and learning to obtain a second instance feature specifically includes: Input the first instance feature into three branches of the feature recombination sub-module for recombination to generate three new feature sequences; Input the three new feature sequences into the MSM sub-module for feature learning, fuse the learned features to obtain key instance features, and input the key instance features into a linear layer for linear transformation, and obtain a second instance feature based on the transformed features and the first instance feature; Input the second instance feature into the instance feature aggregation module of the target model for dimensionality increase and aggregation to obtain a packet-level feature, and input the packet-level feature into the packet prediction module of the target model for classification to obtain the classification result of the slice image to be processed; The step of inputting the second instance feature into the instance feature aggregation module of the target model for dimensionality increase and aggregation to obtain a packet-level feature specifically includes: Input the second instance feature into the linear layer of the instance feature aggregation module of the target model for dimensionality increase to obtain the dimension-increased feature; Use a gated attention mechanism to aggregate the dimension-increased features to obtain an aggregated packet-level feature.

2. The slice image classification method according to claim 1, wherein The step of inputting the first instance feature into three branches of the feature recombination sub-module for recombination to generate three new feature sequences specifically includes: Input the first instance feature into the first branch of the feature recombination sub-module for original scanning to obtain a first new feature sequence that retains the original sequence structure; Input the first instance feature into the second branch of the feature recombination sub-module for grid scanning, reshape to obtain four grid subsequences in different directions, and reorder the four grid subsequences to obtain the recombined second new feature sequence; Input the first instance feature into the third branch of the feature recombination sub-module for hierarchical scanning, reshape to obtain two hierarchical subsequences in different directions, and reorder the two hierarchical subsequences to obtain the recombined third new feature sequence.

3. The slice image classification method according to claim 2, wherein The MSM sub-module includes three branches, where the three branches of the MSM sub-module correspond one-to-one with the three branches of the feature recombination sub-module; Among them, each branch of the MSM sub-module includes a Mamba block and a GCA block. The Mamba block includes a linear layer, a convolutional layer, a SiLU activation function, and a state space model. The GCA block includes layer normalization and an MLP layer.

4. The slice image classification method according to claim 3, characterized in that, Input the three new feature sequences into the MSM sub-module for feature learning, fuse the learned features to obtain the key instance feature, and input the key instance feature into a linear layer for linear transformation. Based on the transformed feature and the first instance feature, obtain the second instance feature, specifically including: Input the three new feature sequences into the three branches of the MSM sub-module respectively, and perform feature learning through the Mamba block and GCA block of each branch to obtain the learning feature of each branch; Fuse the learning features of each branch to obtain the key instance feature, input the key instance feature into a linear layer for linear transformation to obtain the transformed feature, and perform matrix addition on the transformed feature and the first instance feature to obtain the second instance feature.

5. A sliced image classification system, characterized in that, The slice image classification system is applied to the slice image classification method according to any one of claims 1-4. The slice image classification system includes: A slice image acquisition module for acquiring a slice image to be processed, preprocessing the slice image to be processed to obtain a target slice image; A classification model construction module for constructing a slice image classification model, training and testing the slice image classification model to obtain a target model. The target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module, and a packet prediction module; A feature extraction and learning module for inputting the target slice image into the instance feature extraction module of the target model to extract a first instance feature, and inputting the first instance feature into the instance feature learning module of the target model for recombination and learning of the feature to obtain a second instance feature; A feature aggregation and classification module for inputting the second instance feature into the instance feature aggregation module of the target model for dimension elevation and aggregation to obtain a packet-level feature, and inputting the packet-level feature into the packet prediction module of the target model for classification to obtain the classification result of the slice image to be processed.

6. A terminal, characterized in that, The terminal includes: a memory, a processor, and a slice image classification program stored on the memory and executable on the processor. When the slice image classification program is executed by the processor, the steps of the slice image classification method according to any one of claims 1-4 are implemented.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a slice image classification program. When the slice image classification program is executed by a processor, the steps of the slice image classification method according to any one of claims 1-4 are implemented.

Images