Slice image classification method and system, terminal and storage medium

Abstract

The invention discloses a slice image classification method and system, a terminal and a storage medium, and the method comprises the steps: carrying out the preprocessing of a to-be-processed slice image, and obtaining a target slice image; constructing a slice image classification model, training the slice image classification model to obtain a target model, inputting a target slice image into an instance feature extraction module of the target model for feature extraction to obtain a first instance feature, and inputting the first instance feature into an instance feature learning module of the target model for feature recombination and learning to obtain a second instance feature; obtaining a second instance feature; and inputting the second instance feature into an instance feature aggregation module of the target model for dimension raising and aggregation to obtain a packet-level feature, and inputting the packet-level feature into a packet prediction module of the target model for classification to obtain a classification result of the to-be-processed slice image. According to the method, multiple scanning modes are applied to perform feature complementation, key examples are effectively highlighted through global context attention, and accurate classification of slice images is realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a slice image classification method, system, terminal and computer-readable storage medium. Background Art

[0002] Digital slides use a fully automatic microscope scanning system combined with a virtual slide software system to scan and seamlessly stitch traditional glass slides to generate a whole-view digital slide image (WSI). In the practice of pathology, teaching, and research, digital slides have all the functions of traditional slides and have the advantage of not being restricted by space and time. Digital slides are not static pictures. They contain all the lesion information on the glass slide. On a computer, just like under a microscope, different magnifications can be observed, and within a certain range, stepless continuous zooming can be achieved to browse the slides.

[0003] Since full-view digital slice images have high resolution, it is difficult to obtain accurate results by directly analyzing the entire WSI, so many researchers have begun to use multi-instance learning to analyze WSI to achieve tasks such as classification, grading, and survival prediction. However, the target area in WSI accounts for a small proportion, resulting in most multi-instance learning methods based on attention mechanisms being unable to accurately analyze WSI.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a slice image classification method, system, terminal and computer-readable storage medium, aiming to solve the problem that most multi-instance learning methods based on attention mechanism in the prior art cannot accurately classify full-field digital slice images.

[0006] To achieve the above object, the present invention provides a slice image classification method, which comprises the following steps: Acquire a slice image to be processed, and preprocess the slice image to be processed to obtain a target slice image; Constructing a slice image classification model, training and testing the slice image classification model to obtain a target model, wherein the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module and a packet prediction module; Inputting the target slice image into the instance feature extraction module of the target model to extract features to obtain a first instance feature, and inputting the first instance feature into the instance feature learning module of the target model to reorganize and learn features to obtain a second instance feature; The second instance feature is input into the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain a package-level feature, and the package-level feature is input into the package prediction module of the target model for classification to obtain a classification result of the slice image to be processed.

[0007] Optionally, the slice image classification method, wherein the step of inputting the target slice image into the instance feature extraction module of the target model to extract features to obtain a first instance feature, specifically comprises: Inputting the target slice image into the instance feature extraction module for cropping to obtain a plurality of non-overlapping instances, and removing instances that do not contain tissue background from the plurality of instances by a threshold method to obtain a target instance; The target instance is input into a pre-trained ResNet50 model for feature extraction to obtain initial features, and the initial features are input into a multi-layer perceptron for dimensionality reduction to obtain first instance features.

[0008] Optionally, in the slice image classification method, the instance feature learning module comprises: a feature recombination submodule and an MSM submodule; The step of inputting the first instance feature into the instance feature learning module of the target model to perform feature recombination and learning to obtain a second instance feature specifically includes: Inputting the first instance feature into the three branches of the feature recombination submodule for recombination to generate three new feature sequences; The three new feature sequences are input into the MSM submodule for feature learning, the learned features are fused to obtain key instance features, and the key instance features are input into a linear layer for linear transformation, and the second instance features are obtained based on the transformed features and the first instance features.

[0009] Optionally, the slice image classification method, wherein the step of inputting the first instance feature into three branches of the feature recombination module for recombination to generate three new feature sequences, specifically includes: Inputting the first instance feature into the first branch of the feature recombination submodule for original scanning to obtain a first new feature sequence retaining the original sequence structure; Inputting the first instance feature into the second branch of the feature recombination submodule for grid scanning, reshaping to obtain four grid subsequences in different directions, and reordering the four grid subsequences to obtain a reorganized second new feature sequence; The first instance feature is input into the third branch of the feature recombination submodule for hierarchical scanning, and two hierarchical subsequences in different directions are reshaped. The two hierarchical subsequences are reordered to obtain a recombined third new feature sequence.

[0010] Optionally, in the slice image classification method, the MSM submodule includes three branches, wherein the three branches of the MSM submodule correspond one-to-one to the three branches of the feature recombination module; Each branch of the MSM submodule includes a Mamba block and a GCA block, the Mamba block includes a linear layer, a convolutional layer, a SiLU activation function and a state space model, and the GCA block includes layer normalization and an MLP layer.

[0011] Optionally, the slice image classification method, wherein the three new feature sequences are input into the MSM submodule for feature learning, the learned features are fused to obtain key instance features, and the key instance features are input into a linear layer for linear transformation, and the second instance features are obtained based on the transformed features and the first instance features, specifically including: Inputting the three new feature sequences into the three branches of the MSM submodule respectively, performing feature learning through the Mamba block and the GCA block of each branch, and obtaining the learning features of each branch; The learning features of each branch are fused to obtain key instance features, and the key instance features are input into a linear layer for linear transformation to obtain transformed features, and the transformed features are matrix-added with the first instance features to obtain second instance features.

[0012] Optionally, the slice image classification method, wherein the step of inputting the second instance feature into the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain a package-level feature, specifically includes: Inputting the second instance feature into the linear layer of the instance feature aggregation module of the target model for dimension upgrading to obtain a dimension upgraded feature; The gated attention mechanism is used to perform feature aggregation on the dimension-upgraded features to obtain aggregated package-level features.

[0013] In addition, to achieve the above-mentioned purpose, the present invention further provides a slice image classification system, wherein the slice image classification system comprises: A slice image acquisition module is used to acquire a slice image to be processed, and pre-process the slice image to be processed to obtain a target slice image; A classification model building module is used to build a slice image classification model, train and test the slice image classification model to obtain a target model, wherein the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module and a packet prediction module; A feature extraction and learning module, used for inputting the target slice image into the instance feature extraction module of the target model to extract features and obtain a first instance feature, and inputting the first instance feature into the instance feature learning module of the target model to reorganize and learn features and obtain a second instance feature; A feature aggregation classification module is used to input the second instance feature into the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain a package-level feature, and input the package-level feature into the package prediction module of the target model for classification to obtain a classification result of the slice image to be processed.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a slice image classification program stored in the memory and executable on the processor, and when the slice image classification program is executed by the processor, the steps of the slice image classification method as described above are implemented.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a slice image classification program, and when the slice image classification program is executed by a processor, the steps of the slice image classification method described above are implemented.

[0016] In the present invention, the slice image to be processed is preprocessed to obtain the target slice image; a slice image classification model is constructed, the slice image classification model is trained to obtain the target model, the target slice image is input into the instance feature extraction module of the target model for feature extraction to obtain the first instance feature, the first instance feature is input into the instance feature learning module of the target model for feature reorganization and learning to obtain the second instance feature; the second instance feature is input into the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain the package-level feature, the package-level feature is input into the package prediction module of the target model for classification to obtain the classification result of the slice image to be processed. The present invention applies multiple scanning methods for feature complementation, and effectively highlights key instances through global context attention to achieve accurate classification of slice images. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a preferred embodiment of the slice image classification method of the present invention; Figure 2 It is the overall architecture diagram of the target model in the slice image classification method of the present invention; Figure 3 is a schematic diagram of a multi-directional scanning mechanism in the slice image classification method of the present invention; Figure 4 It is a schematic diagram of the GCA block in the slice image classification method of the present invention; Figure 5 is a structural diagram of a preferred embodiment of the slice image classification system of the present invention; Figure 6 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0018] The present application provides a slice image classification method, system and terminal. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0019] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0020] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0021] The slice image classification method described in the preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the slice image classification method includes the following steps: Step S10: Acquire a slice image to be processed, and pre-process the slice image to be processed to obtain a target slice image.

[0022] Specifically, a slice image to be processed (classified) is obtained to preprocess the slice image to be processed. The preprocessing process may include random cropping, data cleaning, data standardization, etc., and finally a target slice image for inputting a model is obtained.

[0023] Step S20, constructing a slice image classification model, training and testing the slice image classification model to obtain a target model, wherein the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module and a packet prediction module.

[0024] Specifically, construct Figure 2 The slice image classification model shown is trained and tested to obtain a target model.

[0025] The slice image classification model is trained and tested, and the method further comprises: Acquire historical slice images and historical classification results corresponding to the historical slice images, use the historical slice images as training samples, use the historical classification results as labels, and construct a data set; The data set is divided into a training set, a test set and a validation set according to a preset ratio. The training set is used to train the slice image classification model, the validation set is used to evaluate the slice image classification model in each round of training, and the test set is used to evaluate the slice image classification model after training.

[0026] In the embodiment, the historical slice images used for training and testing and the historical classification results corresponding to the historical slice images are from two different cancer data sets, namely: 1. Camelyon16 data set for breast cancer lymph node metastasis detection, which contains 240 WSIs from patients with breast cancer lymph node metastasis and 159 WSIs from normal patients. 2. TCGA-Lung data set for cancer subtype classification, which consists of two subtypes, lung squamous cell carcinoma (LUSC) and lung adenocarcinoma (LUAD), with 527 WSIs and 511 WSIs for each subtype, respectively. These two data sets are from public data sets and can be freely downloaded and used.

[0027] Furthermore, in the Camelyon16 dataset, the given training set was randomly divided into a training set and a validation set at a ratio of 4:1, and the test set was used for testing. For the TCGA-Lung dataset, the training set, validation set, and test set were randomly divided at a ratio of 3:1:1 for the experiment. Specifically, the model was trained using the PyTorch framework on a TITAN RTX 4090 GPU with 24GB of memory. During the training process, the initial learning rate was set to , and use The learning rate decay mechanism of the decay coefficient. In addition, in order to optimize the performance of the model, this application uses the Lookahead optimizer (Lookahead is an optimization algorithm that improves the training process of deep learning models by maintaining two sets of weights instead of one set) to optimize the model, and uses the cross entropy loss as the objective function when training the model. At the same time, 200 batches are set for the experiment, and in each batch, the batch size is set to 1 to adapt to the phenomenon that the number of instances cropped from each WSI is different.

[0028] It is understandable that the present application further uses accuracy, sensitivity, precision, F1 score (F1), area under the curve (AUC) and Kappa score (Kappa) as classification performance evaluation criteria. These indicators can evaluate the performance of the slice image classification model from various aspects and ultimately obtain the trained target model.

[0029] Step S30, inputting the target slice image into the instance feature extraction module of the target model for feature extraction to obtain a first instance feature, and inputting the first instance feature into the instance feature learning module of the target model for feature recombination and learning to obtain a second instance feature.

[0030] The step of inputting the target slice image into the instance feature extraction module of the target model to extract features to obtain a first instance feature specifically includes: Inputting the target slice image into the instance feature extraction module for cropping to obtain a plurality of non-overlapping instances, and removing instances that do not contain tissue background from the plurality of instances by a threshold method to obtain a target instance; The target instance is input into a pre-trained ResNet50 model for feature extraction to obtain initial features, and the initial features are input into a multi-layer perceptron for dimensionality reduction to obtain first instance features.

[0031] In this embodiment, given a target slice image, the first step is to crop it into non-overlapping image blocks, called instances, and use the threshold method to remove instances that do not contain tissue background. The size of each instance is 256×256. After that, the ResNet50 model that has been pre-trained on ImageNet is used to extract features from each instance. The dimension of the feature vector of each instance is 1024. In order to reduce feature redundancy and computational overhead, a multi-layer perceptron is used to reduce the feature dimension from 1024 to 256 to obtain the first instance feature.

[0032] Further, the instance feature learning module includes: a feature recombination submodule and an MSM submodule; the instance feature learning module inputting the first instance feature into the target model for feature recombination and learning to obtain the second instance feature specifically includes: The first instance feature is input into the three branches of the feature recombination submodule for recombination to generate three new feature sequences; the three new feature sequences are input into the MSM submodule for feature learning, the learned features are fused to obtain key instance features, and the key instance features are input into a linear layer for linear transformation, and the second instance features are obtained based on the transformed features and the first instance features.

[0033] Further, the inputting the first instance feature into the three branches of the feature recombination module for recombination to generate three new feature sequences specifically includes: Inputting the first instance feature into the first branch of the feature recombination submodule for original scanning to obtain a first new feature sequence retaining the original sequence structure; Inputting the first instance feature into the second branch of the feature recombination submodule for grid scanning, reshaping to obtain four grid subsequences in different directions, and reordering the four grid subsequences to obtain a reorganized second new feature sequence; The first instance feature is input into the third branch of the feature recombination submodule for hierarchical scanning, and two hierarchical subsequences in different directions are reshaped. The two hierarchical subsequences are reordered to obtain a recombined third new feature sequence.

[0034] like Figure 3As shown, it can be understood that the present application reorganizes the extracted features to regenerate a new sequence. During the reorganization process, if the instance length is not enough to divide the scanning direction, the sequence is padded with 0. In particular, the present application retains the original sequence structure in the first branch to capture global feature information, which can make up for the feature loss of the overall sequence caused by the sequence reshaping in the other two branches. In the second branch, a grid scanning strategy is adopted to first divide the sequence into four subsequences in different directions, and then reorder the four subsequences into a new sequence. Since the input features are reorganized, after learning the feature information from the Mamba block, the output features are rearranged to be consistent with the feature order of the first branch. Similarly, in the third branch, a hierarchical scanning strategy is first adopted to reshape the instance features into two subsequences in different directions, and then reorder them into a new sequence. Finally, the output features of the third branch are reordered like the second branch. At the same time, the third branch can also supplement the sequence features of the second branch to a certain extent. Therefore, the three branches of the entire MSM module can better mine the feature associations between sequences.

[0035] Furthermore, the MSM (Multi-scan Mamba) module consists of three branches, each containing a Mamba block and a GCA block. The Mamba block is a deep learning architecture based on the Structured Space Model (SSM) designed to efficiently process long sequence data and is used to effectively model the dependencies between instances, while the GCA (global context attention) block is used to strengthen the relationship between instances and highlight the importance of key instances.

[0036] Specifically, the MSM submodule includes three branches, wherein the three branches of the MSM submodule correspond one-to-one to the three branches of the feature recombination module; wherein each branch of the MSM submodule includes a Mamba block and a GCA block, the Mamba block includes a linear layer, a convolutional layer, a SiLU activation function and a state-space model, and the GCA block includes layer normalization and an MLP layer.

[0037] Furthermore, the three new feature sequences are input into the MSM submodule for feature learning, the learned features are fused to obtain key instance features, and the key instance features are input into a linear layer for linear transformation, and the second instance features are obtained based on the transformed features and the first instance features, specifically including: Inputting the three new feature sequences into the three branches of the MSM submodule respectively, performing feature learning through the Mamba block and the GCA block of each branch, and obtaining the learning features of each branch; The learning features of each branch are fused to obtain key instance features, and the key instance features are input into a linear layer for linear transformation to obtain transformed features, and the transformed features are matrix-added with the first instance features to obtain second instance features.

[0038] The learning process of the MSM submodule is as follows: the new feature sequence is input into the first branch of the MSM submodule for feature learning through the linear layer and the convolutional layer, the extracted features are input into the state space model after the SiLU activation function, and the first Mamba feature is output. Then, the new feature sequence is input into the second branch of the MSM submodule for conversion through the linear layer and the SiLU activation function, and the second Mamba feature is output. Finally, the first learning feature and the second learning feature are matrix multiplied to obtain the target Mamba feature of the corresponding branch in the MSM submodule.

[0039] like Figure 4 As shown in FIG. 1 , the learning process of the GCA block is as follows: the target Mamba features learned from the Mamba branch are subjected to feature fusion and then input into the GCA block. At this time, the input features are first normalized using the layer normalization layer, and their mean is used as the weight. After passing through the MLP layer, the output is matrix multiplied with the original input features to highlight the key instances, and then a linear layer is used to obtain the final output features of the GCA block (i.e., the transformed features). Finally, the transformed features are matrix added with the first instance features to obtain the second instance features.

[0040] Step S40, input the second instance feature to the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain a package-level feature, input the package-level feature to the package prediction module of the target model for classification, and obtain a classification result of the slice image to be processed.

[0041] It should be noted that feature aggregation is a key step in the multi-instance learning method because it enables the model to fully capture the relationships between and within instances. This process is crucial for effectively selecting discriminative instances and accurately predicting the categories of WSI. In the method proposed in this application, multiple MSM modules are used to achieve feature aggregation, so what is obtained here are multiple second instance features.

[0042] Further, the second instance feature is input into the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain a package-level feature, which specifically includes: Inputting the second instance feature into the linear layer of the instance feature aggregation module of the target model for dimension upgrading to obtain a dimension upgraded feature; The gated attention mechanism is used to perform feature aggregation on the dimension-upgraded features to obtain aggregated package-level features.

[0043] In this embodiment, after feature learning by the MSM module, the learned second instance feature is first dimensionally upgraded using a linear layer so that the feature information can be fully expressed. Afterwards, the dimensionally upgraded instance feature is transmitted to the feature aggregation module for feature aggregation. In this process, a general gated attention mechanism is used to realize the aggregation of features to form package-level features. This method can further make the package-level features more inclined to the features of key instances. After the package-level features are formed, a classifier is used to classify the package-level features to achieve the final WSI classification.

[0044] It can be seen that the present invention mainly includes two parts: (1) Mamba module: a multi-directional scanning method is proposed to improve the problem that the original Mamba scanning method cannot effectively mine the spatial information in the sequence. In order to supplement the feature details, a variety of scanning methods are applied to complement the features, fully mine the features, and amplify the importance of key instances. (2): Global context attention module: a GCA module is designed, which can effectively highlight the key instances, allowing the network to better learn the target area in the WSI and achieve accurate classification of the WSI. The experimental results on two public datasets TCGA-Lung and Camelyon16 show that the proposed method is superior to the existing state-of-the-art methods and can better assist pathologists in the classification of pathological sections.

[0045] Furthermore, if Figure 5 As shown, based on the above-mentioned slice image classification method, the present invention also provides a slice image classification system, wherein the slice image classification system includes: The slice image acquisition module 51 is used to acquire the slice image to be processed, and pre-process the slice image to be processed to obtain the target slice image; The classification model building module 52 is used to build a slice image classification model, train and test the slice image classification model to obtain a target model, and the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module and a packet prediction module; A feature extraction and learning module 53 is used to input the target slice image into the instance feature extraction module of the target model to extract features and obtain first instance features, and input the first instance features into the instance feature learning module of the target model to reorganize and learn features and obtain second instance features; The feature aggregation classification module 54 is used to input the second instance feature into the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain a package-level feature, and input the package-level feature into the package prediction module of the target model for classification to obtain a classification result of the slice image to be processed.

[0046] Furthermore, if Figure 6 As shown, based on the above-mentioned slice image classification method and system, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 6 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0047] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a slice image classification program 40 is stored on the memory 20, and the slice image classification program 40 can be executed by the processor 10, thereby realizing the slice image classification method in the present application.

[0048] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, which is used to run the program code or process data stored in the memory 20, such as executing the slice image classification method.

[0049] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0050] In one embodiment, when the processor 10 executes the slice image classification program 40 in the memory 20, the following steps are implemented: Acquire a slice image to be processed, and preprocess the slice image to be processed to obtain a target slice image; Constructing a slice image classification model, training and testing the slice image classification model to obtain a target model, wherein the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module and a packet prediction module; Inputting the target slice image into the instance feature extraction module of the target model to extract features to obtain a first instance feature, and inputting the first instance feature into the instance feature learning module of the target model to reorganize and learn features to obtain a second instance feature; The second instance feature is input into the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain a package-level feature, and the package-level feature is input into the package prediction module of the target model for classification to obtain a classification result of the slice image to be processed.

[0051] The step of inputting the target slice image into the instance feature extraction module of the target model to extract features and obtain the first instance feature specifically includes: Inputting the target slice image into the instance feature extraction module for cropping to obtain a plurality of non-overlapping instances, and removing instances that do not contain tissue background from the plurality of instances by a threshold method to obtain a target instance; The target instance is input into a pre-trained ResNet50 model for feature extraction to obtain initial features, and the initial features are input into a multi-layer perceptron for dimensionality reduction to obtain first instance features.

[0052] Wherein, the instance feature learning module includes: a feature recombination submodule and an MSM submodule; The step of inputting the first instance feature into the instance feature learning module of the target model to perform feature recombination and learning to obtain a second instance feature specifically includes: Inputting the first instance feature into the three branches of the feature recombination submodule for recombination to generate three new feature sequences; The three new feature sequences are input into the MSM submodule for feature learning, the learned features are fused to obtain key instance features, and the key instance features are input into a linear layer for linear transformation, and the second instance features are obtained based on the transformed features and the first instance features.

[0053] The step of inputting the first instance feature into the three branches of the feature recombination module for recombination to generate three new feature sequences specifically includes: Inputting the first instance feature into the first branch of the feature recombination submodule for original scanning to obtain a first new feature sequence retaining the original sequence structure; Inputting the first instance feature into the second branch of the feature recombination submodule for grid scanning, reshaping to obtain four grid subsequences in different directions, and reordering the four grid subsequences to obtain a reorganized second new feature sequence; The first instance feature is input into the third branch of the feature recombination submodule for hierarchical scanning, and two hierarchical subsequences in different directions are reshaped. The two hierarchical subsequences are reordered to obtain a recombined third new feature sequence.

[0054] The MSM submodule includes three branches, wherein the three branches of the MSM submodule correspond one-to-one to the three branches of the feature recombination module; Each branch of the MSM submodule includes a Mamba block and a GCA block, the Mamba block includes a linear layer, a convolutional layer, a SiLU activation function and a state space model, and the GCA block includes layer normalization and an MLP layer.

[0055] The step of inputting the three new feature sequences into the MSM submodule for feature learning, fusing the learned features to obtain key instance features, and inputting the key instance features into a linear layer for linear transformation, and obtaining a second instance feature based on the transformed features and the first instance features, specifically includes: Inputting the three new feature sequences into the three branches of the MSM submodule respectively, performing feature learning through the Mamba block and the GCA block of each branch, and obtaining the learning features of each branch; The learning features of each branch are fused to obtain key instance features, and the key instance features are input into a linear layer for linear transformation to obtain transformed features, and the transformed features are matrix-added with the first instance features to obtain second instance features.

[0056] The step of inputting the second instance feature into the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain a package-level feature specifically includes: Inputting the second instance feature into the linear layer of the instance feature aggregation module of the target model for dimension upgrading to obtain a dimension upgraded feature; The gated attention mechanism is used to perform feature aggregation on the dimension-upgraded features to obtain aggregated package-level features.

[0057] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a slice image classification program, and when the slice image classification program is executed by a processor, the steps of the slice image classification method described above are implemented.

[0058] In summary, the present invention proposes a slice image classification method, system and terminal, the method comprising: obtaining a slice image to be processed, preprocessing the slice image to be processed, and obtaining a target slice image; constructing a slice image classification model, training and testing the slice image classification model, and obtaining a target model, the target model comprising: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module and a package prediction module; inputting the target slice image into the instance feature extraction module of the target model for feature extraction to obtain a first instance feature, inputting the first instance feature into the instance feature learning module of the target model for feature reorganization and learning to obtain a second instance feature; inputting the second instance feature into the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain a package-level feature, inputting the package-level feature into the package prediction module of the target model for classification, and obtaining a classification result of the slice image to be processed. The present invention applies multiple scanning methods for feature complementation, and effectively highlights key instances through global context attention to achieve accurate classification of slice images.

[0059] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0060] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0061] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A slice image classification method, characterized in that: The slice image classification method comprises: Acquire a slice image to be processed, and preprocess the slice image to be processed to obtain a target slice image; Constructing a slice image classification model, training and testing the slice image classification model to obtain a target model, wherein the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module and a packet prediction module; Inputting the target slice image into the instance feature extraction module of the target model to extract features to obtain a first instance feature, and inputting the first instance feature into the instance feature learning module of the target model to reorganize and learn features to obtain a second instance feature; The second instance feature is input into the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain a package-level feature, and the package-level feature is input into the package prediction module of the target model for classification to obtain a classification result of the slice image to be processed.

2. The slice image classification method according to claim 1, characterized in that: The step of inputting the target slice image into the instance feature extraction module of the target model to extract features to obtain a first instance feature specifically includes: Inputting the target slice image into the instance feature extraction module for cropping to obtain a plurality of non-overlapping instances, and removing instances that do not contain tissue background from the plurality of instances by a threshold method to obtain a target instance; The target instance is input into a pre-trained ResNet50 model for feature extraction to obtain initial features, and the initial features are input into a multi-layer perceptron for dimensionality reduction to obtain first instance features.

3. The slice image classification method according to claim 1, characterized in that: The instance feature learning module includes: a feature recombination submodule and an MSM submodule; The step of inputting the first instance feature into the instance feature learning module of the target model to perform feature recombination and learning to obtain a second instance feature specifically includes: Inputting the first instance feature into the three branches of the feature recombination submodule for recombination to generate three new feature sequences; The three new feature sequences are input into the MSM submodule for feature learning, the learned features are fused to obtain key instance features, and the key instance features are input into a linear layer for linear transformation, and the second instance features are obtained based on the transformed features and the first instance features.

4. The slice image classification method according to claim 3, characterized in that: The step of inputting the first instance feature into the three branches of the feature recombination module for recombination to generate three new feature sequences specifically includes: Inputting the first instance feature into the first branch of the feature recombination submodule for original scanning to obtain a first new feature sequence retaining the original sequence structure; Inputting the first instance feature into the second branch of the feature recombination submodule for grid scanning, reshaping to obtain four grid subsequences in different directions, and reordering the four grid subsequences to obtain a reorganized second new feature sequence; The first instance feature is input into the third branch of the feature recombination submodule for hierarchical scanning, and two hierarchical subsequences in different directions are reshaped. The two hierarchical subsequences are reordered to obtain a recombined third new feature sequence.

5. The slice image classification method according to claim 4, characterized in that: The MSM submodule includes three branches, wherein the three branches of the MSM submodule correspond one-to-one to the three branches of the feature recombination module; Each branch of the MSM submodule includes a Mamba block and a GCA block, the Mamba block includes a linear layer, a convolutional layer, a SiLU activation function and a state space model, and the GCA block includes layer normalization and an MLP layer.

6. The slice image classification method according to claim 5, characterized in that: The step of inputting the three new feature sequences into the MSM submodule for feature learning, fusing the learned features to obtain key instance features, and inputting the key instance features into a linear layer for linear transformation, and obtaining a second instance feature based on the transformed features and the first instance features, specifically includes: Inputting the three new feature sequences into the three branches of the MSM submodule respectively, performing feature learning through the Mamba block and the GCA block of each branch, and obtaining the learning features of each branch; The learning features of each branch are fused to obtain key instance features, and the key instance features are input into a linear layer for linear transformation to obtain transformed features, and the transformed features are matrix-added with the first instance features to obtain second instance features.

7. The slice image classification method according to claim 1, characterized in that: The step of inputting the second instance feature into the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain a package-level feature specifically includes: Inputting the second instance feature into the linear layer of the instance feature aggregation module of the target model for dimension upgrading to obtain a dimension upgraded feature; The gated attention mechanism is used to perform feature aggregation on the dimension-upgraded features to obtain aggregated package-level features.

8. A slice image classification system, characterized in that: The slice image classification system comprises: A slice image acquisition module is used to acquire a slice image to be processed, and pre-process the slice image to be processed to obtain a target slice image; A classification model building module is used to build a slice image classification model, train and test the slice image classification model to obtain a target model, wherein the target model includes: an instance feature extraction module, an instance feature learning module, an instance feature aggregation module and a packet prediction module; A feature extraction and learning module, used for inputting the target slice image into the instance feature extraction module of the target model to extract features and obtain a first instance feature, and inputting the first instance feature into the instance feature learning module of the target model to reorganize and learn features and obtain a second instance feature; A feature aggregation classification module is used to input the second instance feature into the instance feature aggregation module of the target model for dimension upgrading and aggregation to obtain a package-level feature, and input the package-level feature into the package prediction module of the target model for classification to obtain a classification result of the slice image to be processed.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a slice image classification program stored in the memory and executable on the processor. When the slice image classification program is executed by the processor, the steps of the slice image classification method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a slice image classification program, and when the slice image classification program is executed by a processor, the steps of the slice image classification method according to any one of claims 1 to 7 are implemented.

Images