Cancer tissue pathology image fine-grained classification method based on SMCNet

Abstract

The invention belongs to the technical field of deep learning image classification, and provides a cancer tissue pathology image fine-grained classification method based on SMCNet. Comprising the following steps: S1, setting an SMCNet network model; s2, collecting a cancer pathology image fine-grained classification public data set, training a classification network based on the formed SMCNet network model, and storing training weights; s3, performing network performance evaluation on the training result of the SMCNet model by utilizing the evaluation indexes of the accuracy, the classification precision, the recall rate, the F1 score and the AUC score; and S4, in combination with the training weight and a result visualization program, reporting a classification prediction result of the test set, and visualizing the classification prediction result by using a thermodynamic diagram. The SMCNet model adopted by the invention has stronger feature extraction capability and high discrimination degree for fine-grained categories, space-channel features are subjected to DRA attention of feature extraction by using a self-attention module, and the design thought of model lightweight is considered on the basis of realizing an optimized feature extraction function.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and image classification technology, specifically relating to a fine-grained classification method for cancer tissue pathology images based on SMCNet. Background Technology

[0002] In the clinical treatment of cancer patients, minimally invasive surgery to sample diseased tissue, prepare slides, and submit them to pathologists for examination is one of the most intuitive, commonly used, and effective methods for determining cancer type. It also provides crucial evaluation criteria for subsequent treatment plans and surgical outcomes. With the development of computer technology and the rise of artificial intelligence, machine learning and neural networks enable computers to learn key features of cancer and surrounding tissues, quickly locating whether a patient has cancer or rapidly identifying the specific type. Using a trained model and relying on computer analysis can also avoid missed or incorrect diagnoses that pathologists might make when analyzing digital slides. In areas with high medical pressure, this method can serve as an auxiliary tool, helping doctors quickly locate cancerous sites, reducing their workload in reading slides, and enabling faster diagnostic results. Furthermore, given the numerous cancer subtypes and the ease with which some early-stage cancers can be confused with ulcers or inflammatory sites, often requiring the reliance on other cumbersome physiological examination indicators to jointly determine whether a patient has cancer, the ease of confusion between cancers and their diverse types, and the laborious nature of manual identification, developing a fine-grained classification model that can accurately extract features and achieve high classification accuracy is particularly important.

[0003] More broadly, in the fields of medical image analysis and clinical pathology diagnosis, the classification and identification of cancer tissue sections and related medical images are the core technical support for achieving early cancer screening and accurate subtyping. Current applications of computer technology in this field mainly include computer statistical analysis, machine learning, deep learning, and association rule classification. Among these, machine learning and deep learning technologies, due to their excellent classification accuracy and environmental adaptability, have become the core focus and mainstream application path of industry research in recent years. Although machine learning methods have been applied to some extent in medical image classification, they have significant technical shortcomings: the complete process requires sequential steps such as image preprocessing, precise annotation of key lesion areas, manual feature extraction, and classifier training. The annotation of key areas, in particular, requires extremely high clinical experience from the operators and must be overseen by senior physicians throughout the process, resulting in high labor costs and long annotation cycles. Furthermore, due to limitations in technical principles, machine learning methods struggle to effectively distinguish some cancer types with symptoms similar to conventional lesions, and cannot deeply explore the intrinsic relationship between lesion characteristics and cancer pathological mechanisms. Model training is also time-consuming. These factors collectively restrict their application effectiveness in clinical mass diagnosis scenarios. Compared to machine learning methods, deep learning technology possesses end-to-end feature learning capabilities. It can autonomously learn common features and accumulate learning experience directly from raw or simply labeled medical image data, effectively overcoming machine learning's reliance on manual annotation and feature extraction, and significantly reducing labor and time costs. Especially in the scenario of fine-grained cancer classification, addressing the challenges of diverse cancer subtypes and the easy confusion of histological features between different subtypes, deep learning-driven fine-grained classification methods, with their stronger feature representation capabilities and more efficient feature mining mechanisms, can accurately capture subtle feature differences between different cancer subtypes, significantly improving classification accuracy. Therefore, they have more prominent technical advantages and clinical application value. Summary of the Invention

[0004] To address the issues of high workload and easy confusion of fine-grained features between categories in cancer identification in the background art, the present invention aims to propose a fine-grained classification method for cancer tissue pathology images based on SMCNet. Compared with the original network, it can improve classification accuracy with a small number of samples and enhance the classification ability of easily confused categories, thus solving the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0006] A fine-grained classification method for cancer tissue pathology images based on SMCNet, characterized by comprising the following steps:

[0007] S1. Set up the SMCNet network model;

[0008] S2. Collect publicly available datasets for fine-grained classification of cancer pathology images, train the classification network based on the formed SMCNet network model, and save the training weights;

[0009] S3. Evaluate the network performance of the SMCNet model training results using evaluation metrics such as accuracy, classification precision, recall, F1 score, and AUC score;

[0010] S4. Combine the training weights with the result visualization program to report the classification prediction results on the test set, and use a heatmap to visualize the classification prediction results.

[0011] In one possible embodiment, step S1 specifically involves the following steps:

[0012] S1-1. Input feature tensor The image is divided into three sizes: small, medium, and large. Each size of patch is then input into a corresponding feature processing branch to extract detail and texture features, balance global and detail features, extract global features, and obtain coarse-grained information about the image category. After padding the tensors of each size patch, the output feature tensors are denoted as follows: , This represents the number of patches to be cut out, and its size is determined by the size of the original input image.

[0013] S1-2. Convert the feature tensors of small, medium, and large sizes. The input attention module is the Scale-Attention Block (SAB), and the output features are denoted as follows: ;

[0014] S1-3. Process the output features After dimensionality reduction and standardization via one-dimensional convolution, feature projections are formed, followed by feature fusion. The input to the network backbone is then optimized using the main loss function Focal Loss, and the output is the prediction error. Simultaneously, As input, a self-supervised attention feature map is generated in the CAM feature supervision branch. The self-supervised features are optimized by the auxiliary loss function CE Loss to output the supervised prediction error. The calculation results of the backbone loss function and the feature supervision loss are weighted to update the model parameters and optimize the final learning result of the model. Finally, the predicted values ​​of each category calculated by the model are passed through the activation function to obtain the final prediction result.

[0015] In one possible embodiment, in S1-1, the input image feature map is... The dimensions were uniformly adjusted to ,Will Cut into The three patch sizes are input into three corresponding feature processing branches to extract detail and texture features, balance global and detail features, extract global features, and obtain coarse-grained information about image categories, respectively. The patch tensors of each size are then processed using padding, and the preliminary feature tensors after processing are denoted as... , This represents the number of patches to be cut out, and its size is determined by the size of the original input image.

[0016] In one possible embodiment, the specific steps of the attention processing module Scale-Attention Block (SAB) in S1-2 are as follows:

[0017] S1-2-1. For the SAB module, there are three sizes: TSAB, MSAB, and LASB, which process the initial feature tensors respectively. The SAB module internally contains an input layer, stem layer, LN layer, convolutional layer, pooling layer, and a feature learning module, SMC Block. For the three sizes of SAB modules, only the input and output feature sizes differ; the internal modules and network layers are the same. The TSAB, MSAB, and LASB modules use... , , The input patch is obtained , , Output of dimensions Batch Size is the training parameter.

[0018] S1-2-2. For the SMC Block in SAB, it contains depthwise separable convolution, LN layer, and DRA attention module for parallel processing of spatial-channel information. Layer Scale and Drop Path are used for optimization before the output features to avoid gradient explosion. The SMC Block has 4 stages in each SAB module, and the number of stacked layers in each stage is 3, 3, 9, and 3.

[0019] S1-2-3. For the DRA module in the SMC Block, its function is to extract the key information contained in three different sized patches; inputting the feature tensors of shapes B, C, H, and W into the DRA yields the following processing: In the channel feature processing branch, channel branch pooling is used, and the spatial dimension is adjusted to 1 by setting dim=[2,3], retaining only the original features of the channel dimension; in the spatial feature processing branch, spatial branch pooling is used, and the channel dimension is adjusted to 1 by setting dim=[1]. Only the original spatial dimension features are retained. The feature tensors that have undergone spatial-channel branch separation are normalized and dropped out before being input into a single-head self-attention module for processing. Subsequently, the processing result of the self-attention module and the original spatial and channel features are weighted separately to obtain the intermediate features after spatial-channel branch processing. The intermediate features are then shape-expanded to restore the dimension of the initial features. Finally, the initial features and intermediate features are weighted to obtain the intermediate features after DRA module processing.

[0020] S1-2-4. The input feature tensors of each SAB module are sequentially processed through a Stem layer for feature dimensionality reduction, an LN layer for tensor normalization, an SMC Block module for feature learning, a one-dimensional convolution with a stride of 1 to expand the number of channels, and Global Average Pooling. Finally, a fully connected Linear layer maps the processed features to the output features. After processing by TSAB, MSAB, and LASB, the output features are denoted as follows: .

[0021] In one possible embodiment, the feature is that step S2 further includes collecting available cancer fine-grained classification datasets to train SMCNet. During training, the AdamW optimizer is used to optimize the network structure parameters, and a breakpoint training method with gradient accumulation is used. Specifically, four rounds of training are conducted, each round consisting of 100 epochs. The gradient is calculated every four epochs, so the weights are only updated after the 4th, 8th, 12th, ..., 100th epochs. Breakpoint training is used to facilitate timely parameter modification based on training results and to avoid training interruptions due to equipment failure. The batch size is set to 32, and the initial learning rate is set to 0.0001. The optimal weights obtained from training the model are recorded in a .pth format file and automatically updated during training. Simultaneously, the loss curve during training is recorded using Tensorboard functions to generate the metric curve during training. After the SMCNet model training is completed, the training weight file is saved.

[0022] In one possible embodiment, the training results and changes in training parameters in step S3 above can be recorded and read by Tensorboard; the model performance is comprehensively evaluated using evaluation metrics such as accuracy, classification precision, recall, F1 score, AUC score, and confidence mAP; since this model involves multi-classification tasks, a balanced weight calculation method is used for the evaluation metrics.

[0023] In one possible embodiment, in step S4, a prediction program is built in a PyTorch environment. The prediction program has two modes: single image prediction, which involves inputting a single image, having the model identify the image, output the probability of the image belonging to each category, reporting the category with the highest probability as the final result, and outputting a heatmap using the model's prediction results; and overall prediction, which involves inputting a set of numbered images taken from a complete WSI image into the network as a folder, with the model reporting the disease types contained in each numbered image based on the prediction results of each image, thus locating the lesion site. The heatmap generation program is built in the PyTorch environment using libraries such as seaborn and OpenCV, and allows setting the category number of the category of interest to determine the category of the key display area of ​​the heatmap.

[0024] Compared with existing technologies, this invention provides a fine-grained classification method for cancer tissue pathology images based on SMCNet, which has the following beneficial effects:

[0025] This invention uses three sizes of patches to locate key information at three levels of granularity, from fine-grained features to coarse-grained features, which respectively help the model retain more texture details and edge information, balance local features and global information, and capture the overall structure of the image.

[0026] This invention proposes a DRA attention method that uses self-attention modules to extract spatial and channel features separately, achieving optimized feature extraction while also taking into account the design concept of lightweight model.

[0027] This invention proposes using different loss functions to calculate parameters for the classification result output branch and the feature supervision branch, and calculating the final output through a double-loss weighted processing method. Simultaneously, it proposes an improved scheme for Focal Loss for multi-classification tasks.

[0028] The SMCNet network proposed in this invention has higher classification accuracy and better performance in evaluation metrics such as F1 score, recall, and classification precision compared to similar lightweight models.

[0029] This invention demonstrates that the SMCNet model possesses stronger feature extraction capabilities and higher discriminative power for fine-grained categories. Furthermore, compared to other feedforward neural networks, SMCNet exhibits better generalization ability, can be applied to other classification datasets, and maintains excellent overall performance. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the fine-grained classification of cancer tissue pathology images based on SMCNet in this invention.

[0031] Figure 2 This is a diagram of the feature processing branch structure using TSAB as an example in this invention;

[0032] Figure 3 This is a structural diagram of the SMC Block in this invention;

[0033] Figure 4 This is a structural diagram of the DRA attention module of the present invention;

[0034] Figure 5 This is a schematic diagram of the SMCNet network structure of the present invention;

[0035] Figure 6 The SMCNet of this invention uses EBHI-Seg to visualize heatmaps. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0038] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of the invention, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations thereof.

[0039] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0040] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0041] Example 1

[0042] A fine-grained classification method for cancer tissue pathology images based on SMCNet, characterized by comprising the following steps:

[0043] S1. Set up the SMCNet network model;

[0044] S1-1. Input feature tensor The image is divided into three sizes: small, medium, and large. Each size of patch is then input into a corresponding feature processing branch to extract detail and texture features, balance global and detail features, extract global features, and obtain coarse-grained information about the image category. After padding the tensors of each size patch, the output feature tensors are denoted as follows: , This represents the number of patches to be cut out, and its size is determined by the size of the original input image.

[0045] S1-2. Convert the feature tensors of small, medium, and large sizes. The input attention module is the Scale-Attention Block (SAB), and the output features are denoted as follows: ;

[0046] S1-3. Process the output features After dimensionality reduction and standardization via one-dimensional convolution, feature projections are formed, followed by feature fusion. The input to the network backbone is then optimized using the main loss function Focal Loss, and the output is the prediction error. Simultaneously, As input, a self-supervised attention feature map is generated in the CAM feature supervision branch. The self-supervised features are optimized by the auxiliary loss function CE Loss to output the supervised prediction error. The calculation results of the backbone loss function and the feature supervision loss are weighted to update the model parameters and optimize the final learning result of the model. Finally, the predicted values ​​of each category calculated by the model are passed through the activation function to obtain the final prediction result.

[0047] Specifically, the attention processing module, Scale-Attention Block (SAB), has three sizes: TSAB, MSAB, and LASB, which are used to process feature tensors of small, medium, and large sizes, respectively. The SAB module internally contains conventional network layers such as LN layers and Global Avg Pooling, as well as the feature learning module SMC Block. The SMC Block contains conventional network layers such as convolutional layers and LN layers, and the Dual-Path Residual Attention (DRA) module for parallel processing of spatial-channel information. DRA is used to extract key information contained in patches of three different sizes. After processing by TSAB, MSAB, and LASB, the output features are denoted as follows: .

[0048] More specifically, the steps of the attention processing module Scale-Attention Block (SAB) are as follows:

[0049] S1-2-1. For the SAB module, there are three sizes: TSAB, MSAB, and LASB, which process the initial feature tensors respectively. The SAB module internally contains an input layer, stem layer, LN layer, convolutional layer, pooling layer, and a feature learning module, SMC Block. For the three sizes of SAB modules, only the input and output feature sizes differ; the internal modules and network layers are the same. The TSAB, MSAB, and LASB modules use... , , Use the input Patch to obtain , , Output of dimensions Batch Size is the training parameter.

[0050] S1-2-2. For the SMC Block in SAB, it contains depthwise separable convolution, LN layer, and DRA attention module for parallel processing of spatial-channel information. Layer Scale and Drop Path are used for optimization before the output features to avoid gradient explosion. The SMC Block has 4 stages in each SAB module, and the number of stacked layers in each stage is 3, 3, 9, and 3.

[0051] S1-2-3. For the DRA module in the SMC Block, its function is to extract the key information contained in three different sizes of patches; inputting the feature tensor of shape (B,C,H,W) into the DRA can achieve the following processing: in the channel feature processing branch, channel branch pooling is used, and the spatial dimension is adjusted to 1 by setting dim=[2,3], only the original features of the channel dimension are retained; in the spatial feature processing branch, spatial branch pooling is used, and the channel dimension is adjusted to 1 by setting dim=[1]. Only the original spatial dimension features are retained. The feature tensors that have undergone spatial-channel branch separation are normalized and dropped out before being input into a single-head self-attention module for processing. Subsequently, the processing result of the self-attention module and the original spatial and channel features are weighted separately to obtain the intermediate features after spatial-channel branch processing. The intermediate features are then shape-expanded to restore the dimension of the initial features. Finally, the initial features and intermediate features are weighted to obtain the intermediate features after DRA module processing.

[0052] S1-2-4. The input feature tensors of each SAB module are sequentially processed through a Stem layer for feature dimensionality reduction, an LN layer for tensor normalization, an SMC Block module for feature learning, a one-dimensional convolution with a stride of 1 to expand the number of channels, and Global Average Pooling. Finally, a fully connected Linear layer maps the processed features to the output features. After processing by TSAB, MSAB, and LASB, the output features are denoted as follows: .

[0053] S1-3. Process the output features After dimensionality reduction and standardization via one-dimensional convolution, feature projections are formed, followed by feature fusion. The input to the network backbone is then optimized using the main loss function Focal Loss, and the output is the prediction error. Simultaneously, As input, a self-supervised attention feature map is generated in the CAM feature supervision branch. The self-supervised features are optimized by the auxiliary loss function CE Loss to output the supervised prediction error. The calculation results of the backbone loss function and the feature supervision loss are weighted to update the model parameters and optimize the final learning result of the model. Finally, the predicted values ​​of each category calculated by the model are passed through the activation function to obtain the final prediction result.

[0054] Specifically, in S1-3, the following will be performed: As input, the features are normalized in shape through one-dimensional convolution to form a feature projection. Then, attention-based feature fusion is performed, and the fused features are optimized using the main loss function Focal Loss, outputting the prediction error. Simultaneously, Input the CAM feature supervision branch; the CAM feature supervision branch generates a self-supervised attention feature map by learning feature projection, optimizes the self-supervised features through the auxiliary loss function CE Loss, and outputs the supervised prediction error; the supervised loss and the main loss are weighted to obtain the total loss, so as to update the model parameters and optimize the final learning result of the model; here, the weighting coefficient of the supervised loss is chosen to be 0.1 to avoid the supervised feature branch from having too much influence on the main branch; the class with the highest predicted value of each class calculated by the model is taken as the final prediction result.

[0055] S2. Collect publicly available datasets for fine-grained classification of cancer pathology images, train the classification network based on the formed SMCNet network model, and save the training weights;

[0056] Specifically, step S2 further includes collecting available fine-grained cancer classification datasets to train SMCNet. During training, the AdamW optimizer is used to optimize the network structure parameters, and a breakpoint training method with gradient accumulation is used. Specifically, four rounds of training are conducted, each round consisting of 100 epochs. The gradient is calculated every 4 epochs, so the weights are only updated after the 4th, 8th, 12th, ..., 100th epochs. Breakpoint training is used to facilitate timely parameter modification based on training results and to avoid training interruptions due to equipment failure. The batch size is set to 32, and the initial learning rate is set to 0.0001. The optimal weights obtained from training the model are recorded in a .pth format file and automatically updated during training. At the same time, the loss curve during training is recorded using Tensorboard functions to generate the indicator curve during training. After the SMCNet model training is completed, the training weight file is saved.

[0057] S3. Evaluate the network performance of the SMCNet model training results using evaluation metrics such as accuracy, classification precision, recall, F1 score, and AUC score;

[0058] Specifically, in step S3 above, the training results and changes in training parameters can be recorded and read by Tensorboard; the model performance is comprehensively evaluated using evaluation metrics such as accuracy, classification precision, recall, F1 score, AUC score, and confidence mAP; since this model involves multi-classification tasks, a balanced weight calculation method is used for the evaluation metrics.

[0059] S4. Combine the training weights with the result visualization program to report the classification prediction results on the test set, and use a heatmap to visualize the classification prediction results.

[0060] Specifically, in step S4, a prediction program is built in the PyTorch environment. The prediction program has two modes: single image prediction, which allows a single image to be input, the model to identify the image, output the probability of the image belonging to each category, report the category with the highest probability as the final result, and output a heatmap using the model's prediction results; and overall prediction, which allows a group of numbered images taken from a complete WSI image to be input into the network in the form of a folder, and the model to report the disease types contained in each numbered image based on the prediction results of each image, thereby locating the lesion site. The heatmap generation program is built in the PyTorch environment using libraries such as seaborn and OpenCV, and allows setting the category number of the category of interest to determine the category of the key display area of ​​the heatmap.

[0061] Example 2

[0062] A fine-grained classification method for cancer tissue pathology images based on SMCNet includes the following steps:

[0063] S1. Construct the SMCNet network model;

[0064] S1-1. Input feature tensor The image is divided into three sizes: small, medium, and large. Each size of patch is then input into a corresponding feature processing branch to extract detail and texture features, balance global and detail features, extract global features, and obtain coarse-grained information about the image category. After padding the tensors of each size patch, the output feature tensors are denoted as follows: , This represents the number of patches to be cut out, and its size is determined by the size of the original input image.

[0065] S1-2. Convert the feature tensors of small, medium, and large sizes. The input attention module is the Scale-Attention Block (SAB), and the output features are denoted as follows: ;

[0066] S1-3. Process the output features After dimensionality reduction and standardization via one-dimensional convolution, feature projections are formed, followed by feature fusion. The input to the network backbone is then optimized using the main loss function Focal Loss, and the output is the prediction error. Simultaneously, As input, a self-supervised attention feature map is generated in the CAM feature supervision branch. The self-supervised features are optimized by the auxiliary loss function CE Loss to output the supervised prediction error. The calculation results of the backbone loss function and the feature supervision loss are weighted to update the model parameters and optimize the final learning result of the model. Finally, the predicted values ​​of each category calculated by the model are passed through the activation function to obtain the final prediction result.

[0067] S2. Collect publicly available datasets for fine-grained classification of cancer pathology images, train the classification network based on the improved SMCNet network model, and save the training weights;

[0068] S3. Evaluate the network performance of the SMCNet model training results using evaluation metrics such as accuracy, classification precision, recall, F1 score, and AUC score;

[0069] S4. Combine the training weights with the result visualization program to report the classification prediction results on the test set, and use a heatmap to visualize the classification prediction results.

[0070] Further, in step S1, the input features are split into... , , The specific implementation method for the patch size step is as follows:

[0071] S1-1-1. Multi-scale segmentation stage, for image input features , Patch sets are generated using sliding windows of different sizes, with three scales set. , The values ​​are 16, 32, and 64 respectively. The sliding step size for each window is... For each scale, the sliding window generates a set of patches using the following formula:

[0072]

[0073] in, , , is a factor used to calculate the number of patches in the height dimension. , is the patch number calculation factor in the width dimension. For input images with the same H and W dimensions, they are numerically equal, and their product is the number of patches at that scale.

[0074] S1-1-2. After the patch is segmented, each patch also needs to undergo Feature Padding to uniformly adjust its H and W dimensions. , , The preliminary features processed in this step are recorded as follows: .

[0075] Furthermore, in steps S1-2, the specific implementation method of the SAB module is as follows:

[0076] S1-2-1. For SAB modules, there are three sizes: TSAB, MSAB, and LASB, which are handled separately. The SAB module internally contains an input layer, a stem layer, an LN layer, convolutional layers, pooling layers, and a feature learning module, the SMC Block. For the three sizes of SAB modules, only the input and output feature sizes differ; the internal modules and network layers are the same. The TSAB, MSAB, and LASB modules use different sizes... , , The input patch is obtained , , Output of dimensions Batch Size is the training parameter.

[0077] S1-2-2. For the SMC Block, its internal structure includes, in sequence: depthwise separable convolution, LN layer, and attention module DRA for parallel processing of spatial-channel information. Layer Scale and Drop Path are used for optimization before the output features to avoid gradient explosion. The SMC Block has 4 stages in each SAB module, and the number of stacked layers in each stage is 3, 3, 9, and 3.

[0078] S1-2-3. For the DRA module in the SMC Block, its function is to extract key information contained in three different sized patches. Inputting the feature tensor of shape (B,C,H,W) into the DRA yields the following processing: In the channel feature processing branch, channel branch pooling is used, and by setting dim=[2,3], the spatial dimension is adjusted to 1, retaining only the original features of the channel dimension; in the spatial feature processing branch, spatial branch pooling is used, and by setting dim=[1], the channel dimension is adjusted to... Only the original spatial dimension features are retained. The feature tensors processed by spatial-channel branch separation are normalized and dropout-operated before being input into a single-head self-attention module for processing. Subsequently, the processing result from the self-attention module and the original spatial and channel features are weighted separately to obtain the intermediate features after spatial-channel branch processing. The intermediate features are then shape-expanded to restore the dimensions of the initial features. Finally, the initial features and intermediate features are weighted to obtain the intermediate features processed by the DRA module.

[0079] S1-2-4. For each input SAB module's feature tensor, the following steps are performed sequentially: feature dimensionality reduction using a Stem layer, tensor normalization using an LN layer, feature learning using an SMC Block module, channel expansion using a one-dimensional convolution with a stride of 1, global average pooling, and finally, mapping the processed features to the output features using a fully connected Linear layer. After processing by TSAB, MSAB, and LASB, the output features are denoted as follows: .

[0080] Further, in steps S1-3, the The classifier's output is obtained by taking the input as follows:

[0081] S1-3-1. Will As input, the features are normalized through one-dimensional convolution to form a feature projection. Then, attention-based feature fusion is performed, and the fused features are used in the classifier backbone for optimization using the Focal Loss function, outputting the prediction error. Because Focal Loss is designed for binary classification tasks, fine-grained tasks often have at least three class labels. Therefore, it is necessary to improve Focal Loss to be suitable for multi-class tasks. This step can be described as follows:

[0082] Focal Loss is a loss function for binary classification problems that address imbalanced sample sizes and the presence of easily learnable sample features. The formula below is for binary classification (labels...) The predicted probability output by the model is ,Right now The probability of this is:

[0083]

[0084] in, , is the focusing parameter; the larger the value, the stronger the suppression of loss for easily classified samples. It is the modulation factor. The modulation factor of easy-to-classify samples is small, so the loss is suppressed, while the modulation factor of difficult-to-classify samples is large, so the loss is preserved.

[0085] definition The predicted probability is the one corresponding to the true category, according to the formula above. It can be defined using the following formula:

[0086]

[0087] The following formula can be used to represent Focal Loss:

[0088]

[0089] Extending Focal Loss to multi-class classification tasks can be achieved through the following methods:

[0090] Suppose the output of a classification model with K labels has the following probability after softmax: , , indicating that the prediction is the first The probability of a class, the true class is Then the predicted probability corresponding to the true class is defined as ,therefore, It is transformed into multi-class Focal Loss, and the calculation method is shown in the following formula:

[0091]

[0092] S1-3-2. In the feature supervision branch, The CAM (Camera-Based Learning) supervised branch is directly input, generating a self-supervised attention feature map. This self-supervised feature map is then optimized using the auxiliary loss function CE Loss, outputting the prediction error of the supervised branch. This error is then weighted with the main loss by a weighting coefficient of 0.1 to obtain the prediction error of the total loss function. The class with the highest probability in the weighted and optimized prediction output is then output as the final prediction result. CE Loss is the cross-entropy loss function, which can be directly implemented using nn.CrossEntropyLoss in PyTorch.

[0093] Furthermore, in S2 above, collecting the dataset and setting the training parameters can be divided into the following steps:

[0094] S2-1. Collect the EBHI-Seg dataset for predicting precancerous colorectal cancer. This dataset is a collection of pathological slides containing colorectal cancer and precancerous lesions, annotated by authoritative experts. It includes six categories: normal, polyps, low-grade intraepithelial neoplasia, high-grade intraepithelial neoplasia, serrated adenoma, and adenocarcinoma. These six categories reflect the developmental correlation of colorectal cancer and are morphologically difficult to distinguish, making them suitable as a fine-grained classification dataset. In the training process, random translation and inversion, along with noise enhancement, are used to improve the robustness of the trained model. Furthermore, before training, the dataset is divided into training, validation, and test sets using the random_split function in a 7:2:1 ratio.

[0095] S2-2. The following settings are made for the training procedure. During training, the AdamW optimizer is used to update network parameters in a timely manner based on the gradient calculated from the loss function. Considering the number of model parameters and the training environment, a breakpoint training method with gradient accumulation is used to train SMCNet. Specifically, four epochs are trained, each epoch consisting of 100 epochs. The gradient is calculated once every four epochs, so the weights are only updated once after the 4th, 8th, 12th, ..., 100th epochs. Breakpoint training is used to facilitate timely parameter modification based on training results and to avoid training interruptions due to equipment failure. The batch size is set to 32, and the initial learning rate is set to 0.0001. The optimal weights obtained from training the model will be recorded in a .pth format file and automatically updated during training. Simultaneously, the parameters during training are recorded using Tensorboard libraries and functions for visualizing the training process and generating accuracy and loss curves.

[0096] Furthermore, in S3 above, the pre-training weights will be saved after the model finishes training, and the training results and changes in training parameters can be recorded and read by Tensorboard; the model performance is comprehensively evaluated using accuracy, classification precision, recall, F1 score, AUC score, and confidence mAP evaluation metrics; since this model involves multi-classification tasks, the evaluation metrics are weighted in a balanced manner.

[0097] Furthermore, in S4, a prediction program is built in the PyTorch environment. The prediction program has two modes: single image prediction, which allows a single image to be input, the model to identify the image, output the probability of the image belonging to each category, report the category with the highest probability as the final result, and output a heatmap using the model's prediction results; and overall prediction, which allows a set of numbered images taken from a complete WSI image to be input into the network in the form of a folder, and the model to report the disease types contained in each numbered image based on the prediction results of each image, thereby locating the lesion site. The heatmap generation program is implemented in the PyTorch environment using related libraries such as seaborn and OpenCV, and allows setting the category number of interest so that the heatmap can selectively display the areas of interest to the user.

[0098] Example 3

[0099] This invention is implemented in the PyTorch deep learning framework. The experimental hardware platform is a computer with a 12th Gen Intel(R) Core (TM) i5-12600f CPU@3.5GHz and an NVIDIA GeForce RTX 4090 (24G) high-performance graphics card.

[0100] like Figure 1As shown, this invention provides a fine-grained classification method for cancer tissue pathology images based on SMCNet, comprising the following steps:

[0101] S1. A method is proposed to partition the input feature tensor into... , , The method for extracting features from patches of varying sizes follows these steps:

[0102] S1-1. Multi-scale segmentation stage, for image input features , Patch sets are generated using sliding windows of different sizes, with three scales set. , The values ​​are 16, 32, and 64 respectively. The sliding step size for each window is... For each scale, the sliding window generates a set of patches using the following formula:

[0103]

[0104] in, , , is a factor used to calculate the number of patches in the height dimension. , is the patch number calculation factor in the width dimension. For input images with the same H and W dimensions, they are numerically equal, and their product is the number of patches at that scale.

[0105] S1-2. After the patch is segmented, each patch also needs to undergo Feature Padding to uniformly adjust its H and W dimensions. , , The preliminary features processed in this step are recorded as follows: .

[0106] S2. Further, the preliminary features are input into SAB modules of three sizes for processing. The specific implementation method of the SAB module is as follows:

[0107] S2-1. The SAB module is divided into three sizes: TSAB, MSAB, and LASB, which handle small, medium, and large patches respectively. The SAB module internally contains an input layer, stem layer, LN layer, convolutional layer, pooling layer, and the feature learning module SMCBlock. For the three SAB module sizes, only the input and output feature sizes differ; the internal modules and network layers are the same. The TSAB, MSAB, and LASB modules respectively enable... , , Use the input Patch to obtain , , Output of dimensions Batch Size is the training parameter.

[0108] S2-2. For SMC Blocks, such as Figure 3 As shown, its internal structure includes, in sequence: depthwise separable convolution, LN layer, and DRA attention module for parallel processing of spatial-channel information. Layer Scale and Drop Path are used for optimization before the output features to avoid gradient explosion. The SMC Block has 4 stages in each SAB module, with 3, 3, 9, and 3 stacked layers in each stage.

[0109] S2-3. The structure of the DRA module in the SMC Block is as follows: Figure 4 As shown. Its function is to extract key information contained in three different sized patches. Inputting the feature tensor of shape (B,C,H,W) into DRA yields the following processing: In the channel feature processing branch, channel branch pooling is used, and the spatial dimension is adjusted to 1 by setting dim = [2,3], retaining only the original features of the channel dimension; In the spatial feature processing branch, spatial branch pooling is used, and the channel dimension is adjusted to 1 by setting dim = [1]. Only the original spatial dimension features are retained. The feature tensors processed by spatial-channel branch separation are normalized and dropout-operated before being input into a single-head self-attention module for processing. Subsequently, the processing result from the self-attention module and the original spatial and channel features are weighted separately to obtain the intermediate features after spatial-channel branch processing. The intermediate features are then shape-expanded to restore the dimensions of the initial features. Finally, the initial features and intermediate features are weighted to obtain the intermediate features processed by the DRA module.

[0110] S2-4. For the feature tensor of each input SAB module, taking TSAB as an example, such as... Figure 2 As shown, the process sequentially involves: feature dimensionality reduction via a STEM layer, tensor normalization via an LN layer, feature learning via an SMC Block module, channel expansion via a one-dimensional convolution with a stride of 1, global average pooling, and finally, mapping the processed features to the output features via a fully connected Linear layer. After TSAB, MSAB, and LASB processing, the output features are denoted as follows: .

[0111] S3. Further, in step S3, will The classifier's output is obtained by taking the input as follows:

[0112] S3-1. Will As input, the features are normalized through one-dimensional convolution to form a feature projection. Then, attention-based feature fusion is performed, and the fused features are used in the classifier backbone for optimization using the FocalLoss loss function, outputting the prediction error. Because FocalLoss is designed for binary classification tasks, fine-grained tasks often have at least three class labels. Therefore, it is necessary to improve FocalLoss to be suitable for multi-class tasks. This step can be described as follows:

[0113] Focal Loss is a loss function for binary classification problems that address imbalanced sample sizes and the presence of easily learnable sample features. It takes the form of the following formula, and is used for binary classification (labeling). The predicted probability output by the model is ,Right now The probability of this is:

[0114]

[0115] in, , is the focusing parameter; the larger the value, the stronger the suppression of loss for easily classified samples. It is the modulation factor. The modulation factor of easy-to-classify samples is small, so the loss is suppressed, while the modulation factor of difficult-to-classify samples is large, so the loss is preserved.

[0116] definition The predicted probability corresponding to the true class can be defined by the following formula:

[0117]

[0118] Focal Loss can be expressed by the following formula:

[0119]

[0120] Extending Focal Loss to multi-class classification tasks can be achieved through the following methods:

[0121] Suppose the output of a classification model with K labels has the following probability after softmax: , , indicating that the prediction is the first The probability of a class, the true class is Then the predicted probability corresponding to the true class is defined as The following is the formula for binary classification Focal Loss. Transformed into multi-class Focal Loss:

[0122]

[0123] S3-2. In the feature supervision branch, The CAM (Camera-Based Observation) supervision branch is directly input, generating a self-supervised attention feature map. This self-supervised feature map is then optimized using the auxiliary loss function CE Loss, outputting the supervised prediction error. This error is weighted with the prediction error calculated from Focal Loss using a weighting coefficient of 0.1 to obtain the model's total loss. 0.1 is a variable parameter; adjusting it alters the influence of the supervised features on the main branch. A larger weighting coefficient means the main branch is more significantly affected by the supervised features. Finally, the category with the highest predicted value is used as the final prediction result. CE Loss is the cross-entropy loss function, which can be directly implemented using nn.CrossEntropyLoss in PyTorch. Combining steps S1 to S3, the SMCNet network model is formed as follows: Figure 5 .

[0124] S4. Further, collect publicly available datasets of fine-grained classification of cancer pathology images, train the classification network based on the formed SMCNet network model, and save the training weights;

[0125] In step S4, the implementation details of collecting the dataset and setting the training parameters are as follows:

[0126] S4-1. Collect the EBHI-Seg dataset for predicting precancerous colorectal cancer. This dataset is a pathological slide dataset including colorectal cancer and precancerous lesions, annotated by authoritative experts. It includes six categories: normal, polyps, low-grade intraepithelial neoplasia, high-grade intraepithelial neoplasia, serrated adenoma, and adenocarcinoma. These six categories reflect the developmental correlation of colorectal cancer and are morphologically difficult to distinguish, making them suitable as a fine-grained classification dataset. In the training procedure, random translation and inversion, along with noise enhancement, are used to improve the robustness of the trained model. Furthermore, before training, the dataset is divided into training, validation, and test sets using the random_split function in a 7:2:1 ratio.

[0127] S4-2. For the training procedure, the following settings are made: The AdamW optimizer is used during training to update network parameters in a timely manner based on the gradient calculated from the loss function. Considering the number of model parameters and the training environment, a breakpoint training method with gradient accumulation is used to train SMCNet. Specifically, four epochs are trained, each epoch consisting of 100 epochs. The gradient is calculated once every four epochs, so the weights are only updated once after the 4th, 8th, 12th, ..., 100th epochs. Breakpoint training is used to facilitate timely parameter modification based on training results and to avoid training interruptions due to equipment failure. The batch size is set to 32, and the initial learning rate is set to 0.0001. The optimal weights obtained from training the model will be recorded in a .pth format file and automatically updated during training. Simultaneously, the parameters during training are recorded using Tensorboard libraries and functions for visualizing the training process and generating accuracy and loss curves.

[0128] S5. Evaluate the network performance of the SMCNet model training results using evaluation metrics such as accuracy, classification precision, recall, F1 score, and AUC score.

[0129] Specifically, in step S5, the calculation method for each evaluation index of the classification network is as follows:

[0130] S5-1. Regarding the confusion matrix, a method for forming a multi-class confusion matrix can be derived by using normalization based on the binary confusion matrix. The calculation methods for other evaluation metrics will be derived from the calculation of the confusion matrix. Taking the generation of the EBHI-Seg test set confusion matrix as an example, it can be achieved through the following steps:

[0131] For a six-class classification problem, its confusion matrix is ​​a matrix of size 1. The confusion matrix is ​​a matrix where each row represents the number of instances of the actual class and each column represents the number of instances of the predicted class. Specifically, each element in the confusion matrix... Indicates that it actually belongs to a category The samples were predicted as class Quantity:

[0132]

[0133] For example, for any one of the six classes in the EBHI-Seg dataset. We can group the other 5 unrelated categories into the same category, i.e., the misclassified category, thus transforming the multi-class problem into a binary classification problem.

[0134] True Positives (TP): Values ​​on the diagonal This indicates that it has been correctly classified into the category. The number of samples;

[0135] False Positives (FP): A column that is not on the diagonal (denoted as the first...). Sum of columns minus diagonal elements ,Right now This indicates that something has been incorrectly categorized. Number of samples in other categories;

[0136] False Negatives (FN): A row that is not on the diagonal (denoted as the first row). The sum of rows minus the diagonal elements ,Right now This indicates the actual category. However, the number of samples that were incorrectly classified into other categories;

[0137] True Negatives (TN): The sum of the entire confusion matrix minus the total number of true positives, false positives, and false negatives for that class. In multi-class classification, true negatives are usually not listed separately because they are computationally complex and their meaning is not as clear as in binary classification, and their importance to model performance evaluation is relatively small.

[0138] S5-2. Based on the confusion matrix, the calculation methods for other evaluation indicators can be derived:

[0139] For a total number of categories C, one of the categories Accuracy refers to the proportion of correctly predicted samples out of the total number of samples. In the case of multi-class classification, accuracy can be defined as:

[0140]

[0141] Precision: Precision refers to the percentage of samples predicted as positive that are actually positive. For multi-class classification problems, precision is calculated separately for each class.

[0142]

[0143] Recall: Recall refers to the percentage of samples that were actually positive that were correctly predicted as positive. Similarly, for multi-class classification problems, recall is calculated separately for each class.

[0144]

[0145] F1 Score: The F1 score is the harmonic mean of precision and recall, providing a comprehensive metric. The F1 score for a multi-class classification model is calculated as follows:

[0146]

[0147] When extending classification tasks from binary to multi-class, the imbalance of sample sizes across classes in the dataset necessitates modifications to the calculation methods for other evaluation metrics in binary classification. To comprehensively assess the performance of the classification model across all classes, multi-class tasks, especially when data distribution across classes is uneven, require the calculation of a weighted average.

[0148] Weighted average is a method of averaging macro-average indicators by weighting the actual number of categories relative to the total number of categories. Weighted average maintains the independence of each category. It is the first The number of samples in a class.

[0149] The weighted average is calculated as follows:

[0150] Weighted average precision:

[0151]

[0152] Weighted average recall rate:

[0153]

[0154] Weighted average F1 score:

[0155]

[0156] To address the issue of uneven sample sizes in the training data, a weighted evaluation metric is used to obtain a better evaluation index. This evaluation index can be calculated in the PyTorch environment by calling relevant functions.

[0157] S6. Combine the training weights with the result visualization program to report the classification prediction results on the test set, and use a heatmap to visualize the classification prediction results.

[0158] Specifically, in step S6, the training weights of the EBHI-Seg dataset are used to predict the category of pathological images and generate a heatmap. The steps are as follows:

[0159] S6-1. In the PyTorch environment, use the os, matplotlib and PIL libraries to build a class prediction program. Input the image path, the program will read the image, convert it into tensor form, and output the prediction output into the trained model. The output result is the class with the highest predicted probability among all classes. Figure 6 This is a test set for EBHI-Seg. One image is randomly selected for each category. The SMCNet and trained weight file are imported into a heatmap generation program based on the seaborn library. This generates a comparison image of the original image, the ground truth label, the CNN model, and the heatmap. A, B, C, and D represent the original image, official label, CNN network comparison result, and SMCNet result for each category. Numbers I to VI represent serrated adenoma, normal tissue, polyp, adenocarcinoma, poorly differentiated intraepithelial neoplasia, and well-differentiated intraepithelial neoplasia, respectively. The program can also number each category from 0 to 5. In the heatmap generation program, by inputting the corresponding number, the program can set the categories the model focuses on for visualized disease screening. Running the heatmap program, by reading the trained model weight file, extracts the features of interest from the input test image based on learned experience. These features are then used to generate a mask image that overlays the original image, completing the heatmap generation. On a heatmap, warmer areas indicate areas of high model attention and are cancerous sites that require our focus. If there are no warmer areas on the heatmap, it means that the image does not contain the type of disease we are interested in.

[0160] S6-2. The heatmap generation program also supports selecting clinically produced WSIs related to disease types and cutting them into images suitable for model reading, for example... , The size and cut sequence number of each image are saved. SMCNet is trained using the cut dataset, and the training weights are saved. The dataset folder can be read by the heatmap generator to report the main disease category (i.e., the type with the highest model prediction accuracy) contained in each image in the folder as the cancer type for that image.

[0161] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0162] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0163] Although embodiments of the present invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions, and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents. The preferred embodiments of the present invention have been specifically described above, but the invention is not limited to the described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the invention, and all such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A fine-grained classification method for cancer tissue pathology images based on SMCNet, characterized in that: Includes the following steps: S1. Set up the SMCNet network model; S2. Collect publicly available datasets for fine-grained classification of cancer pathology images, train the classification network based on the formed SMCNet network model, and save the training weights; S3. Evaluate the network performance of the SMCNet model training results using evaluation metrics such as accuracy, classification precision, recall, F1 score, and AUC score; S4. Combine the training weights with the result visualization program to report the classification prediction results on the test set, and use a heatmap to visualize the classification prediction results.

2. The fine-grained classification method for cancer tissue pathology images based on SMCNet according to claim 1, characterized in that: The specific steps of step S1 are as follows: S1-1. Input feature tensor The image is divided into three sizes: small, medium, and large. Each size of patch is then input into a corresponding feature processing branch to extract detail and texture features, balance global and detail features, extract global features, and obtain coarse-grained information about the image category. After padding the tensors of each size patch, the output feature tensors are denoted as follows: , This represents the number of patches to be cut out, and its size is determined by the size of the original input image. S1-2. Convert the feature tensors of small, medium, and large sizes. The input attention module is the Scale-Attention Block (SAB), and the output features are denoted as follows: ; S1-3. Process the output features After dimensionality reduction and standardization via one-dimensional convolution, feature projections are formed, followed by feature fusion. The input to the network backbone is then optimized using the main loss function Focal Loss, and the output is the prediction error. Simultaneously, As input, a self-supervised attention feature map is generated in the CAM feature supervision branch. The self-supervised features are optimized by the auxiliary loss function CE Loss to output the supervised prediction error. The calculation results of the backbone loss function and the feature supervision loss are weighted to update the model parameters and optimize the final learning result of the model. Finally, the predicted values ​​of each category calculated by the model are passed through the activation function to obtain the final prediction result.

3. The fine-grained classification method for cancer tissue pathology images based on SMCNet according to claim 1, characterized in that: In step S1-1, the input image feature map The dimensions were uniformly adjusted to ,Will Cut into The three patch sizes are input into three corresponding feature processing branches to extract detail and texture features, balance global and detail features, extract global features, and obtain coarse-grained information about image categories, respectively. The patch tensors of each size are then processed using padding, and the preliminary feature tensors after processing are denoted as... , This represents the number of patches to be cut out, and its size is determined by the size of the original input image.

4. The fine-grained classification method for cancer tissue pathology images based on SMCNet according to claim 1, characterized in that: In S1-2, the specific steps of the attention processing module Scale-Attention Block (SAB) are as follows: S1-2-1. For the SAB module, there are three sizes: TSAB, MSAB, and LASB, which process the initial feature tensors respectively. The SAB module internally contains an input layer, stem layer, LN layer, convolutional layer, pooling layer, and a feature learning module, SMC Block. For the three sizes of SAB modules, only the input and output feature sizes differ; the internal modules and network layers are the same. The TSAB, MSAB, and LASB modules use... , , The input patch is obtained , , Output of dimensions Batch Size is the training parameter. S1-2-2. For the SMC Block in SAB, it contains depthwise separable convolution, LN layer, and DRA attention module for parallel processing of spatial-channel information. Layer Scale and Drop Path are used for optimization before the output features to avoid gradient explosion. The SMC Block has 4 stages in each SAB module, and the number of stacked layers in each stage is 3, 3, 9, and 3. S1-2-3. For the DRA module in the SMC Block, its function is to extract the key information contained in three different sizes of patches; inputting the feature tensor of shape (B,C,H,W) into the DRA can achieve the following processing: in the channel feature processing branch, channel branch pooling is used, and the spatial dimension is adjusted to 1 by setting dim=[2,3], only the original features of the channel dimension are retained; in the spatial feature processing branch, spatial branch pooling is used, and the channel dimension is adjusted to 1 by setting dim=[1]. Only the original spatial dimension features are retained. The feature tensors that have undergone spatial-channel branch separation are normalized and dropped out before being input into a single-head self-attention module for processing. Subsequently, the processing result of the self-attention module and the original spatial and channel features are weighted separately to obtain the intermediate features after spatial-channel branch processing. The intermediate features are then shape-expanded to restore the dimension of the initial features. Finally, the initial features and intermediate features are weighted to obtain the intermediate features after DRA module processing. S1-2-4. The input feature tensor of each SAB module must pass through the stem layer for feature dimensionality reduction, the LN layer for tensor normalization, the SMC Block module for feature learning, the one-dimensional convolution with a stride of 1 to expand the number of channels, the global average pooling, and finally the fully connected linear layer to map the processed features to the output features. After processing by TSAB, MSAB, and LASB, the output features are denoted as follows: .

5. The fine-grained classification method for cancer tissue pathology images based on SMCNet according to claim 1, characterized in that: Step S2 further includes collecting available cancer fine-grained classification datasets to train SMCNet. During training, the AdamW optimizer is used to optimize the network structure parameters. A breakpoint training method with gradient accumulation is used, specifically training in four rounds, each round consisting of 100 epochs. The gradient is calculated every 4 epochs, so the weights are only updated after the 4th, 8th, 12th, ..., 100th epochs. Breakpoint training is used to facilitate timely parameter modification based on training results and to avoid training interruptions due to equipment failure. The batch size is set to 32, and the initial learning rate is set to 0.0001. The optimal weights obtained from training the model are recorded in a .pth format file and automatically updated during training. At the same time, the loss curve during training is recorded using Tensorboard functions to generate the indicator curve during training. After the SMCNet model training is completed, the training weight file is saved.

6. The fine-grained classification method for cancer tissue pathology images based on SMCNet according to claim 1, characterized in that: In step S3, the training results and changes in training parameters can be recorded and read by Tensorboard; the model performance is comprehensively evaluated using accuracy, classification precision, recall, F1 score, AUC score, and confidence mAP evaluation metrics; since this model involves multi-classification tasks, a balanced weight calculation method is used for the evaluation metrics.

7. The fine-grained classification method for cancer tissue pathology images based on SMCNet according to claim 1, characterized in that: In step S4, a prediction program is built in the PyTorch environment. The prediction program has two modes: single image prediction and overall prediction. Single image prediction involves inputting a single image, the model identifying the image, outputting the probability of the image belonging to each category, and reporting the category with the highest probability as the final result. A heatmap is also output using the model's prediction results. Overall prediction involves inputting a set of numbered images taken from a complete WSI image into the network in the form of a folder. The model will report the disease types contained in each numbered image based on the prediction results of each image, thereby locating the lesion site. The heatmap generation program is built in the PyTorch environment using libraries such as seaborn and OpenCV. The category of interest number is set to determine the category of the key display area of ​​the heatmap.

Images