Lung cancer tissue classification method based on improved Swinin-Transformer
By using an improved Swin-Transformer network and a multi-instance learning method, the challenge of lung cancer tissue classification in high-resolution full-view pathological images was solved, achieving efficient and accurate lung cancer tissue identification without the need for professional annotation.
Patent Information
- Application Number
- CN202510881525.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-27
- Publication Date
- 2025-11-11
AI Technical Summary
Existing technologies struggle to directly process high-resolution full-view pathological images and effectively classify lung cancer tissues, especially due to the small size and uneven distribution of cancerous areas, which makes classification difficult and professional pathological annotation is costly.
An improved Swin-Transformer network is used for feature extraction, combining multi-instance learning and self-supervised contrastive learning. Classification is performed through image-level labels. Multi-scale features are extracted using a multi-head attention mechanism and an efficient multi-scale attention module with cross-space learning. The model is trained and evaluated using Focal loss.
Without the need for professional annotation, it improves the accuracy and robustness of lung cancer tissue classification, effectively identifies cancer types and lesion areas, and reduces annotation costs.
Smart Images
Figure CN120931982A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of deep learning and image classification technology, specifically relating to a lung cancer tissue classification method based on an improved Swin-Transformer. Background Technology
[0002] In clinical practice, pathological diagnosis is the "gold standard" for definitively identifying lung cancer subtypes. Because lung pathological images often contain very small cancerous areas, and may even include multiple metastases, it is difficult for doctors to reach consistent diagnostic results. With the rapid development of medical imaging technology and the establishment of medical databases, leveraging artificial intelligence to empower healthcare has gradually become a research hotspot in the field of computer vision. Applying deep learning methods to the classification of lung cancer pathological images can improve the diagnostic efficiency of pathologists. Deep learning technology can quickly identify lung cancer subtypes, which will greatly improve the efficiency and quality of pathological image diagnosis. Furthermore, it can avoid subjective errors by doctors, such as misdiagnosis or missed diagnosis due to differences in experience and fatigue. A well-trained computer-aided diagnostic system can use the same standards to analyze the morphological features of cells and intercellular structures in the images, avoiding interference from human factors and achieving higher objectivity and accuracy.
[0003] Currently, using deep learning for classification and diagnosis of full-view pathological images faces two main challenges. First, full-view images have extremely high resolution, typically reaching 100,000 × 1,000,000 pixels, making them unsuitable for direct input into deep learning networks. Second, due to the large data volume, cancer pathological sections often contain some normal tissue, requiring professional pathologists to annotate the lesion areas—a time-consuming, labor-intensive, and costly process. Therefore, how to classify and diagnose full-view pathological images without professional pathological annotation is a major current challenge. Summary of the Invention
[0004] (a) Technical problems to be solved
[0005] To address the shortcomings of existing technologies, this invention provides a lung cancer tissue classification method based on an improved Swin-Transformer. This method employs a multi-instance learning approach, relying solely on image-level labels to achieve full-view image classification, and solves the problem that the small and uneven distribution of cancerous areas in lung cancer tissue makes it difficult to identify the type of cancer and the lesion area.
[0006] (II) Technical Solution
[0007] To achieve the above objectives, the present invention specifically adopts the following technical solution:
[0008] A lung cancer tissue classification method based on an improved Swin-Transformer includes the following steps:
[0009] S1. Prepare the dataset: Collect images of full-field pathological tissue sections of the lungs to construct a dataset, including tissue sections of various lung cancer subtypes and normal lung tissue sections;
[0010] S2. Dataset preprocessing: Divide the dataset into training and testing sets, and cut it into patches at two different magnification levels. Filter out invalid backgrounds and holes in the patches, and normalize the colors of the remaining patches.
[0011] S3. Feature Extraction: Improve the Swin-Transformer network framework as a feature extractor, and use a contrastive learning method to extract the features of the patch under two magnification levels.
[0012] S4. Feature Fusion Training: The features extracted from the two patches at different magnifications are fused together and trained using a multi-instance learning network to obtain the predicted label for the test image.
[0013] S5. Result Evaluation: Compare the predicted labels with the true labels to generate a confusion matrix, and evaluate the results using three metrics: accuracy, F1-score, and ROC curve.
[0014] S6. Results Display: A sliding window is used to scan the full-view image, and a heat map is generated for each scan patch to display the cancer area on the entire full-view image.
[0015] Furthermore, in step S2, the preprocessing steps for the dataset are as follows:
[0016] S2-1. Divide the dataset into a training set and a test set in a 7:3 ratio;
[0017] S2-2. Extract the images of the pathological slides at two magnification levels, "5×" and "10×", and cut them into 224×224 patches. Record the position information of each patch in the pathological slide.
[0018] S2-3. Filter out blank backgrounds and holes, i.e., the non-blank area is less than 80% of the patch;
[0019] S2-4. Perform color normalization on the remaining patches using the Vahadane normalization method.
[0020] Furthermore, in step S3, the specific steps of feature extraction are as follows:
[0021] S3-1. Improve the feature extraction network by introducing a multi-head attention mechanism into the input layer of the Swin-Transformer network and using a residual structure to connect the input and output. Add an efficient multi-scale attention module (EMA) with cross-space learning between each Swin TransformerBlock layer of the network to enhance the ability to extract multi-scale features;
[0022] S3-2. Using the improved Swing Transformer as the feature extractor, a self-supervised contrastive learning approach is adopted to extract features from the patch. Slices of each lung cancer subtype are recorded as positive examples, and slices of normal lung tissue are recorded as negative examples. The model is trained by generating positive and negative sample pairs and minimizing the NT-Xent loss, so that the network can learn the effective features of the data. Feature extraction is performed on slices under two different magnifications.
[0023] Furthermore, in step S4, the specific steps for feature fusion training are as follows:
[0024] S4-1. Based on the positional encoding of the patch, a weighted aggregation method is used to fuse features at different scales at the corresponding positions. The feature fusion formula is as follows:
[0025]
[0026] in λ represents the fused features, m represents the number of high-magnification images, and λ represents the weight of the low-magnification features (adjusted according to the slice size). Indicates high magnification characteristics. Indicates low magnification characteristics;
[0027] S4-2. Focal loss is used as the loss function in the model for training at both the patch and bag levels to balance the imbalance between positive and negative examples. The type of the entire WSI image is inferred from the features and labels of multiple instances. The loss function formula is as follows:
[0028] FL(p t )=-a t (1-p t ) γ log(p t )
[0029] In the formula: p t To predict the probability. t p represents the sample weights. γ is the focusing parameter. t This reflects the degree of closeness to the actual category label, p tThe larger the value, the closer it is to the actual category label, and the more accurate the classification.
[0030] Furthermore, in step S5, the model's results were evaluated using a variety of quantitative evaluation metrics, including accuracy, F1 score, and ROC curve, to quantitatively analyze the classification results.
[0031] Furthermore, in step S6, the results are displayed by scanning the entire field-of-view image under test layer by layer, with a window size of 224×224 and an overlap of 50%. A precise heatmap for each patch is generated by calculating the gradient input during backpropagation, and finally fed back to the entire WSI image. In the heatmap, the deeper the heatmap of the region corresponding to each patch, the greater the probability that the region is a lesion.
[0032] (III) Beneficial Effects
[0033] Compared with existing technologies, this invention provides a lung cancer tissue classification method based on an improved Swin-Transformer, which has the following beneficial effects:
[0034] In the feature extraction stage, this invention employs a self-supervised contrastive learning method, which calculates the loss by comparing positive and negative samples, thereby distinguishing lesion areas from normal tissue areas. This allows for the extraction of lesion area information from images without lesion area annotations.
[0035] This invention employs an improved Swin-Transformer as a feature extractor for multi-scale patch feature extraction, enhancing the connections between different parts of the image. The Swin-Transformer incorporates an efficient multi-scale attention module (EMA) for cross-spatial learning, which can effectively fuse features at different scales during hierarchical training of the network, thereby improving the model's receptive field while enhancing its accuracy and robustness.
[0036] In the model training phase, this invention employs a multi-instance learning approach, weighting and fusing the extracted data before inputting it into a multi-instance learning network for training. This process identifies the key patches that are most discriminative for the classification task, thereby obtaining bag-level representation vectors. The classifier then performs predictions on the full-view pathological images. Attached Figure Description
[0037] Figure 1 This invention relates to the lung cancer tissue classification method based on the improved Swin-Transformer.
[0038] Figure 2 This is a schematic diagram of the improved Swin-Transformer structure described in this invention;
[0039] Figure 3 This is a schematic diagram of the lung cancer tissue classification method based on the improved Swing-Transformer described in this invention. Detailed Implementation
[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0041] Example
[0042] This invention is implemented in the PyTorch deep learning framework. The experimental hardware platform is a 16vCPU Intel(R)Xeon(R)Platinum 8352V CPU@2.10GHz and an RTX4090 (24GB).
[0043] like Figure 1 As shown, this invention provides a lung cancer tissue classification method based on an improved Swin-Transformer, comprising the following steps:
[0044] Furthermore, in step S1, full-field pathological sections (WSIs) of lung tissue are collected from TCGA and TCIA. These sections are prepared by staining with hematoxylin and eosin solution. Each WSI typically contains billions of pixels and records complete histopathological information.
[0045] Furthermore, in step S2, the preprocessing steps for the dataset are as follows:
[0046] S2-1. Slice the dataset at "5×" and "10×" magnifications, with each slice being a non-overlapping 224×224 patch. Filter out slices with blank backgrounds and effective areas less than 80%. Encode the position of each slice, ensuring that slices at the same location correspond at different magnifications. Use the `random_split()` function to randomly divide the entire dataset into training and test sets at a ratio of 70% and 30%. Each lung cancer tissue digital pathology image is treated as a multi-example package, with each multi-example containing several unlabeled examples. Pathology images of various lung cancer subtypes are designated as positive examples, and normal lung tissue slices are designated as negative examples.
[0047] S2-2. The Vahadane color normalization algorithm was used to normalize the sliced images. Color normalization corrects colors to the same color space, enhancing the robustness of the training model and improving the staining quality of cell nuclei and cytoplasm. The color normalization algorithm maps patches corresponding to valid tissues to the same color space, ensuring that the phenotypic distribution of patches with the same tissue structure follows a normal distribution. This eliminates the potential impact of inconsistencies in lymph node pathological images caused by uncontrollable differences such as different slide acquisition instruments and staining depths. The normalization formula is shown below:
[0048]
[0049] Where T is the transformation matrix, X ref X src It is the color space feature of the source image and the reference image, μ ref μ src These are the mean values of the source image and the reference image, respectively.
[0050] Furthermore, in step S3, the feature extraction using the improved Swin-Transformer is performed as follows:
[0051] S3-1. Modify the input layer of the Swin-Transformer network model by fusing the original PatchPartitioning and Linear Embedding modules, introducing a multi-head attention mechanism, and using a residual structure to connect the input and output into a Multi-Head Embedding module. This module segments the input image into fixed-size blocks, treating the pixel values of each block as elements of a vector, ultimately forming a two-dimensional vector matrix to represent the entire image. Add an EMA attention module between each Swin TransformerBlock layer of the network. This module transforms all channels into corresponding dimensions and then segments these dimensions to form sub-feature groups, each with corresponding spatial semantic features. This module employs three parallel channels to extract the weights of each set of features. The first two paths are 1×1 pooling layers, which use one-dimensional global average pooling operations in the X and Y directions to obtain information from the encoded channels in the two directions respectively. The third path is a 3×3 convolutional layer, which captures multi-scale feature representations through 3×3 convolution, replacing the original GN normalization and average pooling operations, thereby enhancing the ability to extract multi-scale features.
[0052] S3-2. The dataset used in this study only contains labeled images of the entire field of view, without any manual annotation. The data employs a self-supervised learning method based on contrastive learning, using an improved Swin-Transformer as the feature extractor. The model is trained by generating positive and negative sample pairs and minimizing the NT-Xent loss. Slices of each lung cancer subtype are recorded as positive examples, and slices of normal lung tissue are recorded as negative examples, enabling the network to learn effective features of the lesion region.
[0053] S3-3. In step 2, the dataset is sliced at magnification levels of "5×" and "10×". The slices at "5×" are designated as low-magnification slices, and the slices at "10×" are designated as high-magnification slices. Feature extraction is performed on the two sets of images, and the extracted weights are named "patch_low" and "patch_high" respectively. This method is used to learn the mapping relationship between slices at different magnification levels and bag labels in the training dataset.
[0054] S3-4. Set specific parameters for the improved Swin-Transformer network structure: input image size imgsize = 224×224, training batch size batch_size = 64, training iterations Epoch = 100, learning rate lr = 0.0002, decay = 0.00001, label smoothing = 0.1, and the ADAM optimizer is used. Place the improved Swin-Transformer network structure with the set parameters into a computer with the configured environment and use the lung cancer tissue patch-level images in the training set extracted in step S2 for training.
[0055] Furthermore, in step S4, the specific steps for feature fusion training are as follows:
[0056] S4-1. Based on the pyramid mechanism in WSI, the feature vectors extracted in step 4 at different magnification levels are concatenated according to positional encoding, and then input into an attention-based multi-instance learning pooling network to achieve feature fusion of feature vectors. The prediction formula for the feature fusion label is as follows:
[0057]
[0058] in λ represents the fused features, m represents the number of high-magnification images, and λ represents the weight of the low-magnification features (adjusted according to the slice size). Indicates high magnification characteristics. Indicates low magnification characteristics;
[0059] S4-2. Focalloss is chosen as the loss function for the feature fusion training module, primarily to address the classification inaccuracies caused by imbalanced data. Its expression is as follows:
[0060] FL(p t )=-a t (1-p t ) γ log(p t )
[0061] In the formula: p t To predict the probability. t p represents the sample weights. γ is the focusing parameter. t This reflects the degree of closeness to the actual category label, p t The larger the value, the closer it is to the actual category label, and the more accurate the classification.
[0062] Furthermore, in step S5, after training is completed, the trained optimal model is used to predict the pathological images of lung cancer tissue to be tested, and the classification results are obtained.
[0063] To accurately evaluate the classification performance of breast tissue pathology images, this study employed a series of quantitative evaluation indicators, including accuracy, F1 score, and ROC curve, to quantify the classification results. The relevant calculation formulas are as follows:
[0064]
[0065] TP: Represents the number of positive samples that were misclassified as positive samples; FN: Represents the number of positive samples that were misclassified as negative samples; FP: Represents the number of negative samples that were misclassified as positive samples; TN: Represents the number of negative samples that were misclassified as negative samples.
[0066]
[0067] In the formula, Precision is the precision rate and Recall is the recall rate.
[0068]
[0069] In the formula, s i ,s j : These are the model prediction scores for the i-th positive sample and the j-th negative sample, respectively. i >s j When, f(s) i ,s j The value is 1 if it is not 1, otherwise the value is 0.
[0070] Further, in step S6, the image at 5x magnification is first extracted from the full-view pathological image to be tested. A 224×224 window is used for layer-by-layer scanning to obtain patches at relative positions. The window overlap is 50%. An accurate heatmap for each patch is generated by calculating the gradient input during backpropagation. The overlap is calculated using a weighted method, and finally, the entire WSI image is fed back. In the heatmap, the deeper the heatmap of the region corresponding to each patch, the greater the probability that the region is a lesion, thus achieving lesion region detection in the entire pathological image.
[0071] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A lung cancer tissue classification method based on an improved Swin-Transformer, characterized in that: The following testing steps are included: S1. Prepare the dataset: Collect images of full-field pathological tissue sections of the lungs to construct a dataset, including tissue sections of various lung cancer subtypes and normal lung tissue sections; S2. Dataset preprocessing: Divide the dataset into training and test sets, and cut it into patches at two different magnification levels. Filter out invalid backgrounds and holes in the patches, and normalize the colors of the remaining patches. S3. Feature Extraction: Improve the Swin-Transformer network framework as a feature extractor, and use a contrastive learning method to extract the features of the patch under two magnification levels; S4. Feature Fusion Training: The features extracted from the two different magnifications of the patch are fused together and trained in conjunction with a multi-instance learning network to obtain the predicted label of the test image. S5. Result Evaluation: The predicted labels are compared with the true labels to generate a confusion matrix, which is then evaluated using three metrics: accuracy, F1-score, and ROC curve. S6. Results Display: A sliding window is used to scan the full-view image, and a heat map is generated for each scan patch to display the cancer area on the entire full-view image.
2. The lung cancer tissue classification method based on the improved Swing-Transformer according to claim 1, characterized in that, The preprocessing of the dataset in S2 specifically includes: S2-1. Divide the dataset into a training set and a test set in a 7:3 ratio; S2-2. Extract the images of the pathological slides at two magnifications, "5×" and "10×", and cut them into 224×224 patches. Record the position information of each patch in the pathological slide. S2-3. Filter out blank backgrounds and holes, i.e., the non-blank area is less than 80% of the patch; S2-4. Perform color normalization on the remaining patches using the Vahadane normalization method.
3. The lung cancer tissue classification method based on the improved Swing-Transformer according to claim 1, characterized in that: In step S3, the specific steps of feature extraction are as follows: S3-1. Improve the feature extraction network by introducing a multi-head attention mechanism into the input layer of the Swing-Transformer network and using a residual structure to connect the input and output; add an efficient multi-scale attention module (EMA) with cross-space learning between each Swing TransformerBlock layer of the network to enhance the ability to extract multi-scale features. S3-2. Using the improved Swing Transformer as the feature extractor, a self-supervised contrastive learning approach is adopted to extract features from the patch. Slices of each lung cancer subtype are recorded as positive examples, and slices of normal lung tissue are recorded as negative examples. The model is trained by generating positive and negative sample pairs and minimizing the NT-Xent loss, so that the network can learn the effective features of the data. Feature extraction is performed on slices under two different magnifications.
4. The lung cancer tissue classification method based on the improved Swing-Transformer according to claim 1, characterized in that: In step S4, the specific steps for multi-instance feature fusion training are as follows: S4-1. Based on the positional encoding of the patch, a weighted aggregation method is used to fuse features at different scales at the corresponding positions. The feature fusion formula is as follows: in This represents the fused features, where m represents the number of high-magnification images, and λ represents the weight of the low-magnification features, adjusted according to the slice size. Indicates high magnification characteristics. Indicates low magnification characteristics; S4-2. Focal loss is used as the loss function in the model for training at both the real patch level and bag level. The type of the entire WSI image is inferred by backpropagating the features and labels of multiple instances. The loss function formula is as follows: FL(p t )=-a t (1-p t ) γ log(p t ) In the formula: p t For predicting probabilities; a t γ is the sample weight; p is the focusing parameter; γ is the focusing factor; p is the sample weight. t This reflects the degree of closeness to the actual category label, p t The larger the value, the closer it is to the actual category label, and the more accurate the classification.
5. The lung cancer tissue classification method based on the improved Swing-Transformer according to claim 1, characterized in that: The S5 uses a variety of quantitative evaluation indicators, including accuracy, F1 score, and ROC curve, to evaluate the classification results quantitatively.
6. The lung cancer tissue classification method based on the improved Swing-Transformer according to claim 1, characterized in that: In step S6, the full-view image to be tested is scanned layer by layer in a window with a size of 224×224 and an overlap of 50%. The precise heatmap of each patch is generated by calculating the gradient input during backpropagation, and finally fed back to the whole WSI image. In the heatmap, the deeper the heatmap of the area corresponding to each patch, the greater the probability that the area is a lesion area.