Lung cancer pathological image classification method based on vision transformer model
Patent Information
- Application Number
- CN202410726911.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-06
- Publication Date
- 2026-10-13
AI Technical Summary
对于肺癌而言，肺癌治疗手段众多，但是肺癌Ⅲ期及以上五年生存率不足10％
[0044](1)系统运行时，获取肺癌病理学图像，并对其进行数据预处理，得到训练图像数据集和测试图像数据集，随机处理图像，构建基于Vision Transformer的特征提取器后输入全连接层学习特征并分类，根据数据集各类别肺癌病理图像总数量，初步确定模型训练次数，获取不同时期和不同类别肺癌的诊疗康复措施，将诊疗康复和肺癌类别一一对应，存储在PostgreSQL数据库中，先将待检测肺癌病理图像进行图像预处理，对转换得到的张量格式数据逐通道进行标准化处理，送入训练好的模型中进行预测，从数据库模块中提取出相应所属类别的建议治疗措施，评估模型在不同数据集上的表现。
Smart Images
Figure CN122906407A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image classification technology, specifically to a lung cancer pathological image classification method based on the Vision Transformer model. Background Technology
[0002] Pathological diagnosis is the foundation of individualized treatment for patients, and histopathological sections are the "gold standard" for lung cancer diagnosis. While there are numerous treatment options for lung cancer, the five-year survival rate for stage III and above is less than 10%. Therefore, effective prediction and timely diagnosis of lung cancer are crucial for increasing the likelihood of early cure and survival rates.
[0003] Currently, lung cancer pathological diagnosis is mainly conducted manually, resulting in low levels of automation and long diagnostic times. Furthermore, due to factors such as compensation, workload, and training cycles, there is a severe shortage of pathologists. my country has approximately 0.12 pathologists per 10,000 people, far lower than the 0.52 in the United States and 0.55 in Europe, respectively. This cannot meet the growing clinical demand for lung cancer diagnosis, leading to a heavy burden and responsibility on pathologists, and increasing the rates of misdiagnosis and missed diagnosis.
[0004] In recent years, with the continuous development of artificial intelligence, deep learning has demonstrated its powerful capabilities in the field of computer vision. Deep learning technology can assist pathologists in detecting lung cancer, and experts and scholars have conducted extensive research in this area, achieving certain research results, such as EfcientNet, ResNet, and RacNet. Therefore, many scholars have adopted deep learning neural networks for the detection and identification of lung cancer. However, these methods still have problems such as feature information loss and lack of multi-scale information in extracting deep features of lung cancer, and there is still room for improvement in predictive performance. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a lung cancer pathological image classification method based on the Vision Transformer model, which solves the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides the following technical solution: a lung cancer pathological image classification method based on the Vision Transformer model, comprising the following steps:
[0007] Step 1: Obtain lung cancer pathology images and perform data preprocessing to obtain training image datasets and test image datasets;
[0008] Step 2: Randomly crop the image to different sizes and aspect ratios, randomly flip the image horizontally, convert the image to PyTorch tensor format, and standardize the tensor data;
[0009] Step 3: First, build an image segmentation module and a linear transformation module, then build a feature extractor based on Vision Transformer, and finally build a fully connected layer as a classifier. The image is first segmented into blocks and then linearly mapped. The linear mapping result is then fed into the feature extractor. After the feature extractor extracts features, it is input into the fully connected layer to learn the features and classify them.
[0010] Step 4: Based on the total number of lung cancer pathology images in each category of the dataset, preliminarily determine the number of training iterations for the model;
[0011] Step 5: Save and test the model. Save the trained model to the hard drive and test the model using the test set in the dataset.
[0012] Step Six: Build a database to obtain diagnosis, treatment and rehabilitation measures for lung cancer at different stages and in different categories, and store the diagnosis, treatment and rehabilitation measures in a PostgreSQL database in a one-to-one correspondence with lung cancer categories;
[0013] Step 7: First, perform image preprocessing on the pathological images of lung cancer to be detected. Starting from the center of each image, crop along both sides and convert the processed images into a data type format specific to deep learning. Standardize the tensor format data channel by channel. Feed the processed images into the trained model for prediction. The model sends the prediction results into the database module for data matching. Then, extract the suggested treatment measures for the corresponding category from the database module and output them together with the detection classification results.
[0014] Step 8: By dividing the dataset into training and validation sets and alternating between training and validation, the model's performance on different datasets is evaluated, increasing data diversity. Data augmentation techniques, including rotation, scaling, and flipping, are used to generate more training samples, improving the model's robustness and stability. The model is then fine-tuned in detail by combining the prediction results of multiple models, and error cases are analyzed.
[0015] Preferably, a large amount of high-quality lung cancer pathology image data is acquired, and the data sources are diversified, including lung cancer images from different patients and different pathological types, and the data contains clearly labeled categories.
[0016] Preferably, for the acquired lung cancer pathology image training dataset, the given image is first randomly cropped into different sizes and aspect ratios, and then the cropped image is scaled to a specified size of 224×224. Next, each cropped image is randomly rotated horizontally with a 50% probability. Third, the image processed above is converted into tensor format of the deep learning framework PyTorch. Fourth, the converted tensor format data is standardized channel by channel, and the mean and standard deviation after standardization are both 0.5.
[0017] Preferably, an image segmentation module and a linear transformation module are constructed first, then a feature extractor based on Vision Transformer is constructed, and finally a fully connected layer is constructed as a classifier. The image is first segmented into blocks and then linearly mapped. The linear mapping result is then fed into the feature extractor. After the feature extractor extracts the features, it is input into the fully connected layer to learn the features and classify them.
[0018] Vision Transformer-based feature extractors include Transformer encoders;
[0019] The Transformer encoder obtains the encoding module by repeating it L times. The encoding module mainly consists of the following parts: normalization layer, multi-head attention layer and fully connected layer.
[0020] The Transformer encoder takes the linearly transformed image and the region code obtained after block division as input, performs normalization processing first, and then sends the processed image to the multi-head attention layer to obtain the contribution of each region in the image to the final classification task. Based on the contribution, weights are assigned to the corresponding regions. Finally, after summarizing all feature information, the data is normalized again and sent to the fully connected neural network for learning.
[0021] Dividing the encoder into multiple heads creates multiple subspaces, allowing the encoder to focus on different aspects of information. The multi-head attention layer is obtained by repeating the scaled dot product attention model h times and merging the outputs.
[0022] The preferred formula for calculating the scaled dot product attention model is: The scaling dot product attention calculation process is as follows:
[0023] (1) The normalized input image is represented by X = [X1, X2, ..., XT] to represent T block regions. For each region, three different vectors are obtained by multiplying it with three different linear transformation matrices. Here, the three vectors are named Q, K, and V respectively; Q = WqX, K = WkX, V = WvX, Wq, Wk, and Wv are linear transformation matrices, and d represents the dimension of Q and K.
[0024] (2) Calculate the attention score, which is the contribution weight of each region to the final classification task. The result is the attention score by performing a dot product operation between the Q matrix and the K matrix of each part.
[0025] (3) The attention score calculated above is scaled down proportionally to make the model have a more stable gradient, and the result is normalized by Softmax.
[0026] (4) Multiply the scores by V to obtain a weighted feature information, and add these feature information together to obtain the final output tensor z of the algorithm.
[0027] Preferably, the number of training iterations for the model is initially determined based on the total number of lung cancer pathology images in each category of the dataset;
[0028] The number of different types of lung cancer pathology images in the statistical dataset is counted. Based on the size and complexity of the dataset, the number of training iterations is initially determined. The appropriate batch size is selected based on the performance during training. During training, data augmentation techniques, including rotation, scaling, flipping, and color transformation, are used to increase data diversity.
[0029] Preferably, the preprocessed classification dataset is fed into the model for training. During the model training process, the number of iterations is set to 100, the batch size is 32, and the learning rate is 0.001. Through multiple iterations and backpropagation, the model parameters are updated to improve the classification accuracy of the model.
[0030] Preferably, based on the category of different lung cancer pathology images, the hazards and corresponding recommended treatment and rehabilitation measures are constructed;
[0031] Recommended treatment measures:
[0032] Grade 1: Early-stage lung cancer
[0033] Predictive characteristics: The tumor is small, confined to the lungs, and has not metastasized to lymph nodes or distant sites.
[0034] Recommended treatment: The patient is in good health and surgical removal of the tumor is recommended. Surgical types include wedge resection, lobectomy, or pneumonectomy, depending on the location and size of the tumor. For patients with a risk of recurrence after surgery, local radiotherapy is recommended to reduce the risk of recurrence. Regular CT scans and blood tests are required after surgery to monitor for recurrence or metastasis.
[0035] Grade 2: Locally Advanced Lung Cancer
[0036] Predictive characteristics: The tumor is large, invading surrounding tissues or having local lymph node metastasis, but no distant metastasis has occurred.
[0037] Recommended treatment: Chemotherapy should be administered before surgery to shrink the tumor size. The chemotherapy regimen includes platinum-based combination therapy. After neoadjuvant chemotherapy, if the tumor has shrunk and the patient's health permits, surgical resection should be performed. Postoperative chemotherapy or radiotherapy should be administered to eliminate residual cancer cells and reduce the risk of recurrence.
[0038] Grade 3: Advanced or metastatic lung cancer
[0039] Predictive characteristics: The tumor has spread to distant organs or other parts of the body, including bones, liver, and brain;
[0040] Recommended treatment measures: For patients with severe symptoms, systemic chemotherapy is recommended to control tumor progression and alleviate symptoms. Chemotherapy regimens include platinum-based combination therapy and the use of immune checkpoint inhibitors to enhance the patient's immune system's ability to attack cancer cells, including nivolumab and pembrolizumab. For patients with severe and incurable symptoms, palliative care is provided to relieve pain and other symptoms. Palliative care includes analgesics, radiation therapy to relieve symptoms, nutritional support, and psychological counseling.
[0041] Preferably, the pathological images of lung cancer to be detected are first preprocessed, that is, each image is scaled sequentially by reducing the smallest side to 256 and the other side to the same scale. Then, each image is cropped from the center along both sides, and the cropped image size is 224×224. Third, the processed images are converted into tensor format using the deep learning framework PyTorch. Fourth, the converted tensor format data is standardized channel by channel, and the mean and standard deviation after standardization are both 0.5. The processed images are then fed into a trained model for prediction. The model sends the prediction results to the database module for data matching. Subsequently, the suggested treatment measures corresponding to the category are extracted from the database module and output together with the detection classification results.
[0042] The lung cancer pathology image classification method based on the Vision Transformer model uses cross-validation to evaluate the model's generalization ability during training and use, prevents overfitting, adjusts the model's hyperparameters, learning rate, batch size, and number of Transformer layers, collects new lung cancer pathology image data to enrich the training dataset, improves the model's robustness and accuracy, and specifically improves the model structure and training strategy.
[0043] This invention provides a lung cancer pathological image classification method based on the Vision Transformer model, which has the following beneficial effects:
[0044] (1) During system operation, lung cancer pathological images are acquired and preprocessed to obtain training image datasets and test image datasets. Images are randomly processed, and a feature extractor based on Vision Transformer is constructed and input into a fully connected layer to learn features and classify them. Based on the total number of lung cancer pathological images in each category of the dataset, the number of training times of the model is initially determined. Diagnosis and rehabilitation measures for different periods and different categories of lung cancer are acquired, and the diagnosis and rehabilitation measures are matched one-to-one with the lung cancer categories and stored in the PostgreSQL database. The lung cancer pathological images to be detected are first preprocessed, and the tensor format data obtained by conversion is standardized channel by channel and sent to the trained model for prediction. The suggested treatment measures of the corresponding category are extracted from the database module, and the performance of the model on different datasets is evaluated.
[0045] (2) Global Self-Attention Mechanism: Traditional neural networks (CNNs) use the image as the entire input and employ convolutional and pooling layers to extract local features. When extracting features in convolutional layers, larger kernels are selected to obtain a larger receptive field, thus acquiring more image information and global features. However, large kernels lead to a surge in computation, hindering model depth and reducing computational performance. The Vision Transformer model, on the other hand, uses a global self-attention mechanism to automatically learn relevant features in the image and dynamically adjusts its receptive field according to the image size. It calculates attention weights between each pair of image patches to aggregate global information, ensuring that each image patch possesses information from any other image patch, thus providing a flexible and powerful method for spatial relationship modeling. Therefore, the Vision Transformer model can handle images of arbitrary sizes, without being limited by a fixed input size. It achieves higher accuracy in lung cancer pathological image detection and classification.
[0046] (3) By utilizing the characteristics of deep learning, there is no need to manually extract image features. You only need to feed the image data into the model for training to obtain the pathological image features of each category of lung cancer in the dataset.
[0047] (4) After detecting the lung cancer pathological image category, the present invention sends it to the database for matching and corresponding suggested treatment measures, and finally outputs the suggested measures and the detection classification results together. Attached Figure Description
[0048] Figure 1 This is a schematic diagram of the steps of the lung cancer pathological image classification method based on the Vision Transformer model of the present invention;
[0049] Figure 2This is a flowchart illustrating the lung cancer pathological image classification method based on the Vision Transformer model of the present invention.
[0050] Figure 3 This is an overview of the lung cancer pathological image detection and analysis module model of the lung cancer pathological image classification method based on the Vision Transformer model of this invention;
[0051] Figure 4 This is the encoder block for the lung cancer pathological image classification method based on the Vision Transformer model of this invention;
[0052] Figure 5 This is the MLP block of the lung cancer pathological image classification method based on the Vision Transformer model of this invention;
[0053] Figure 6 This invention relates to a multi-head attention method for lung cancer pathological image classification based on the Vision Transformer model. Detailed Implementation
[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0055] Example 1
[0056] This invention provides a lung cancer pathological image classification method based on the Vision Transformer model. Please refer to [link / reference]. Figure 1 This includes the following steps:
[0057] Step 1: Obtain lung cancer pathology images and perform data preprocessing to obtain training image datasets and test image datasets;
[0058] Step 2: Randomly crop the image to different sizes and aspect ratios, randomly flip the image horizontally, convert the image to PyTorch tensor format, and standardize the tensor data;
[0059] Step 3: First, build an image segmentation module and a linear transformation module, then build a feature extractor based on Vision Transformer, and finally build a fully connected layer as a classifier. The image is first segmented into blocks and then linearly mapped. The linear mapping result is then fed into the feature extractor. After the feature extractor extracts features, it is input into the fully connected layer to learn the features and classify them.
[0060] Step 4: Based on the total number of lung cancer pathology images in each category of the dataset, preliminarily determine the number of training iterations for the model;
[0061] Step 5: Save and test the model. Save the trained model to the hard drive and test the model using the test set in the dataset.
[0062] Step Six: Build a database to obtain diagnosis, treatment and rehabilitation measures for lung cancer at different stages and in different categories, and store the diagnosis, treatment and rehabilitation measures in a PostgreSQL database in a one-to-one correspondence with lung cancer categories;
[0063] Step 7: First, perform image preprocessing on the pathological images of lung cancer to be detected. Starting from the center of each image, crop along both sides and convert the processed images into a data type format specific to deep learning. Standardize the tensor format data channel by channel. Feed the processed images into the trained model for prediction. The model sends the prediction results into the database module for data matching. Then, extract the suggested treatment measures for the corresponding category from the database module and output them together with the detection classification results.
[0064] Step 8: By dividing the dataset into training and validation sets and alternating between training and validation, the model's performance on different datasets is evaluated, increasing data diversity. Data augmentation techniques, including rotation, scaling, and flipping, are used to generate more training samples, improving the model's robustness and stability. The model is then fine-tuned in detail by combining the prediction results of multiple models, and error cases are analyzed.
[0065] Example 2
[0066] This embodiment is an explanation based on Embodiment 1. Please refer to it. Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 Specifically, this involves acquiring a large amount of high-quality lung cancer pathology image data and ensuring that the data sources are diverse, including lung cancer images from different patients and different pathological types, with clearly labeled categories.
[0067] For the acquired lung cancer pathology image training dataset, the given images are first randomly cropped to different sizes and aspect ratios, and then the cropped images are scaled to a specified size of 224×224. Next, each cropped image is randomly rotated horizontally with a 50% probability. Third, the images processed above are converted into tensor format of the deep learning framework PyTorch. Fourth, the converted tensor format data is standardized channel by channel, and the mean and standard deviation after standardization are both 0.5.
[0068] First, we construct an image segmentation module and a linear transformation module. Then, we construct a feature extractor based on Vision Transformer. Finally, we construct a fully connected layer as a classifier. The image is first segmented into blocks and then linearly mapped. The linear mapping result is then fed into the feature extractor. After the feature extractor extracts the features, it is input into the fully connected layer to learn the features and classify them.
[0069] Vision Transformer-based feature extractors include Transformer encoders;
[0070] The Transformer encoder obtains the encoding module by repeating it L times. The encoding module mainly consists of the following parts: normalization layer, multi-head attention layer and fully connected layer.
[0071] The Transformer encoder takes the linearly transformed image and the region code obtained after block division as input, performs normalization processing first, and then sends the processed image to the multi-head attention layer to obtain the contribution of each region in the image to the final classification task. Based on the contribution, weights are assigned to the corresponding regions. Finally, after summarizing all feature information, the data is normalized again and sent to the fully connected neural network for learning.
[0072] Dividing the encoder into multiple heads creates multiple subspaces, allowing the encoder to focus on different aspects of information. The multi-head attention layer is obtained by repeating the scaled dot product attention model h times and merging the outputs.
[0073] In this step, for the obtained lung cancer pathology image test dataset, each image is scaled sequentially by reducing the smallest side to 256 and scaling the other side by the same ratio. Next, each image is cropped from the center along both sides, resulting in a cropped image size of 224×224. Third, the processed images are converted into a deep learning-specific data type format, such as the tensor format of the PyTorch deep learning framework in this example. Fourth, the converted tensor format data is standardized channel by channel, resulting in a mean and standard deviation of 0.5.
[0074] Example 3
[0075] This embodiment is an explanation based on Embodiment 1. Please refer to it. Figure 1 Specifically, the calculation formula for the scaled dot product attention model is: The scaling dot product attention calculation process is as follows:
[0076] (1) The normalized input image is represented by X = [X1, X2, ..., XT] to represent T block regions. For each region, three different vectors are obtained by multiplying it with three different linear transformation matrices. Here, the three vectors are named Q, K, and V respectively; Q = WqX, K = WkX, V = WvX, Wq, Wk, and Wv are linear transformation matrices, and d represents the dimension of Q and K.
[0077] (2) Calculate the attention score, which is the contribution weight of each region to the final classification task. The result is the attention score by performing a dot product operation between the Q matrix and the K matrix of each part.
[0078] (3) The attention score calculated above is scaled down proportionally to make the model have a more stable gradient, and the result is normalized by Softmax.
[0079] (4) Multiply the scores by V to obtain a weighted feature information, and add these feature information together to obtain the final output tensor z of the algorithm.
[0080] Based on the total number of lung cancer pathology images in each category of the dataset, the number of training iterations for the model was initially determined.
[0081] The number of different types of lung cancer pathology images in the statistical dataset is counted. Based on the size and complexity of the dataset, the number of training iterations is initially determined. The appropriate batch size is selected based on the performance during training. During training, data augmentation techniques, including rotation, scaling, flipping, and color transformation, are used to increase data diversity.
[0082] In this embodiment, diverse training samples are generated through preprocessing steps such as random cropping, horizontal flipping, conversion to PyTorch tensor format, and standardization. This avoids model overfitting and improves the model's adaptability to different images. Data augmentation techniques such as rotation, scaling, flipping, and color transformation are added during training to further increase data diversity and enhance the model's robustness to different data distributions. Linear transformations are performed on image blocks to effectively capture local information, preserve the spatial structure features of the image, and improve the model's sensitivity to detailed information. Utilizing the multi-head attention mechanism of the Transformer encoder, multiple normalization processes and attention calculations are performed to fully explore the contribution of different regions in the image to the classification task, forming multiple subspaces that focus on different aspects of information, thereby improving feature extraction performance. Based on the size and complexity of the dataset, a reasonable number of training iterations is initially determined and adjusted during training based on performance, selecting an appropriate batch size to ensure the model achieves optimal performance within a limited time.
[0083] Example 4
[0084] This embodiment is an explanation based on Embodiment 1. Please refer to it. Figure 1 Specifically, the preprocessed classification dataset is fed into the model for training. During the model training process, the number of iterations is set to 100, the batch size is 32, and the learning rate is 0.001. Through multiple iterations and backpropagation, the model parameters are updated to improve the classification accuracy of the model.
[0085] Based on the different categories of lung cancer pathology images, construct the hazards and corresponding recommended treatment and rehabilitation measures;
[0086] Recommended treatment measures:
[0087] Grade 1: Early-stage lung cancer
[0088] Predictive characteristics: The tumor is small, confined to the lungs, and has not metastasized to lymph nodes or distant sites.
[0089] Recommended treatment: The patient is in good health and surgical removal of the tumor is recommended. Surgical types include wedge resection, lobectomy, or pneumonectomy, depending on the location and size of the tumor. For patients with a risk of recurrence after surgery, local radiotherapy is recommended to reduce the risk of recurrence. Regular CT scans and blood tests are required after surgery to monitor for recurrence or metastasis.
[0090] Grade 2: Locally Advanced Lung Cancer
[0091] Predictive characteristics: The tumor is large, invading surrounding tissues or having local lymph node metastasis, but no distant metastasis has occurred.
[0092] Recommended treatment: Chemotherapy should be administered before surgery to shrink the tumor size. The chemotherapy regimen includes platinum-based combination therapy. After neoadjuvant chemotherapy, if the tumor has shrunk and the patient's health permits, surgical resection should be performed. Postoperative chemotherapy or radiotherapy should be administered to eliminate residual cancer cells and reduce the risk of recurrence.
[0093] Grade 3: Advanced or metastatic lung cancer
[0094] Predictive characteristics: The tumor has spread to distant organs or other parts of the body, including bones, liver, and brain;
[0095] Recommended treatment measures: For patients with severe symptoms, systemic chemotherapy is recommended to control tumor progression and alleviate symptoms. Chemotherapy regimens include platinum-based combination therapy and the use of immune checkpoint inhibitors to enhance the patient's immune system's ability to attack cancer cells, including nivolumab and pembrolizumab. For patients with severe and incurable symptoms, palliative care is provided to relieve pain and other symptoms. Palliative care includes analgesics, radiation therapy to relieve symptoms, nutritional support, and psychological counseling.
[0096] In this embodiment, based on the predicted level of lung cancer pathological image classification, corresponding treatment strategies and rehabilitation measures are formulated, providing personalized medical plans, optimizing treatment effects, and improving patients' survival rates and quality of life. By matching treatment recommendations with classification results and combining them with relevant medical literature and clinical research results in the database, the scientific validity and feasibility of the recommendations are ensured.
[0097] Example 5
[0098] For a lung cancer pathology image classification method based on the Vision Transformer model, please refer to [link / reference]. Figure 2 Specifically: First, the pathological images of lung cancer to be detected are preprocessed, that is, each image is scaled to 256 on its smallest side, and the other side is scaled by the same proportion. Second, each image is cropped from the center along both sides, and the cropped image size is 224×224. Third, the processed images are converted into tensor format using the deep learning framework PyTorch. Fourth, the converted tensor format data is standardized channel by channel, and the mean and standard deviation after standardization are both 0.5. The processed images are then fed into a trained model for prediction. The model sends the prediction results to the database module for data matching. Subsequently, the suggested treatment measures corresponding to the category are extracted from the database module and output together with the detection classification results.
[0099] During model training and use, cross-validation was used to evaluate the model's generalization ability and prevent overfitting. The model's hyperparameters, learning rate, batch size, and number of Transformer layers were adjusted. New lung cancer pathological image data were collected to enrich the training dataset, improve the model's robustness and accuracy, and the model structure and training strategy were improved in a targeted manner.
[0100] In this embodiment, the smallest side of the image is scaled to 256, and the other side is scaled proportionally. Then, it is cropped from the image center to a size of 224×224 to ensure consistent image size input to the model, reducing interference from image size variations and improving the model's generalization ability. The image data is converted to PyTorch tensor format and standardized channel-by-channel to ensure consistent data distribution characteristics (mean and standard deviation both 0.5) when input to the model, improving training stability and convergence speed. The multi-head attention mechanism and Transformer encoder structure of the Vision Transformer model are used to accurately extract global and local features of the image. Through image segmentation and linear transformation, the original image information is converted into a form suitable for Transformer processing, enabling the model to more effectively focus on important features in different regions of the image and improve classification accuracy.
[0101] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A lung cancer pathological image classification method based on the Vision Transformer model, characterized by: Includes the following steps: Step 1: Obtain lung cancer pathology images and perform data preprocessing to obtain training image datasets and test image datasets; Step 2: Randomly crop the image to different sizes and aspect ratios, randomly flip the image horizontally, convert the image to PyTorch tensor format, and standardize the tensor data; Step 3: First, build an image segmentation module and a linear transformation module, then build a feature extractor based on Vision Transformer, and finally build a fully connected layer as a classifier. The image is first segmented into blocks and then linearly mapped. The linear mapping result is then fed into the feature extractor. After the feature extractor extracts features, it is input into the fully connected layer to learn the features and classify them. Step 4: Based on the total number of lung cancer pathology images in each category of the dataset, preliminarily determine the number of training iterations for the model; Step 5: Save and test the model. Save the trained model to the hard drive and test the model using the test set in the dataset. Step Six: Build a database to obtain diagnosis, treatment and rehabilitation measures for lung cancer at different stages and in different categories, and store the diagnosis, treatment and rehabilitation measures in a PostgreSQL database in a one-to-one correspondence with lung cancer categories; Step 7: First, perform image preprocessing on the pathological images of lung cancer to be detected. Starting from the center of each image, crop along both sides and convert the processed images into a data type format specific to deep learning. Standardize the tensor format data channel by channel. Feed the processed images into the trained model for prediction. The model sends the prediction results into the database module for data matching. Then, extract the suggested treatment measures for the corresponding category from the database module and output them together with the detection classification results. Step 8: By dividing the dataset into training and validation sets and alternating between training and validation, the model's performance on different datasets is evaluated, increasing data diversity. Data augmentation techniques, including rotation, scaling, and flipping, are used to generate more training samples, improving the model's robustness and stability. The model is then fine-tuned in detail by combining the prediction results of multiple models, and error cases are analyzed.
2. The lung cancer pathological image classification method based on the Vision Transformer model according to claim 1, characterized in that: Acquire a large amount of high-quality lung cancer pathology image data, and ensure that the data sources are diverse, including lung cancer images from different patients and different pathological types, with clearly labeled categories.
3. The lung cancer pathological image classification method based on the VisionTransformer model according to claim 1, characterized in that: For the acquired lung cancer pathology image training dataset, the given images are first randomly cropped to different sizes and aspect ratios, and then the cropped images are scaled to a specified size of 224×224. Next, each cropped image is randomly rotated horizontally with a 50% probability. Third, the images processed above are converted into tensor format of the deep learning framework PyTorch. Fourth, the converted tensor format data is standardized channel by channel, and the mean and standard deviation after standardization are both 0.
5.
4. The lung cancer pathological image classification method based on the VisionTransformer model according to claim 1, characterized in that: First, we build an image segmentation module and a linear transformation module. Then, we build a feature extractor based on VisionTransformer. Finally, we build a fully connected layer as a classifier. The image is first segmented into blocks and then linearly mapped. The linear mapping result is then fed into the feature extractor. After the feature extractor extracts the features, it is fed into the fully connected layer to learn the features and classify them. VisionTransformer-based feature extractors include Transformer encoders; The Transformer encoder obtains the encoding module by repeating it L times. The encoding module mainly consists of the following parts: normalization layer, multi-head attention layer and fully connected layer. The Transformer encoder takes the linearly transformed image and the region code obtained after block division as input, performs normalization processing first, and then sends the processed image to the multi-head attention layer to obtain the contribution of each region in the image to the final classification task. Based on the contribution, weights are assigned to the corresponding regions. Finally, after summarizing all feature information, the data is normalized again and sent to the fully connected neural network for learning. Dividing the encoder into multiple heads creates multiple subspaces, allowing the encoder to focus on different aspects of information. The multi-head attention layer is obtained by repeating the scaled dot product attention model h times and merging the outputs.
5. The lung cancer pathological image classification method based on the VisionTransformer model according to claim 4, characterized in that: The scaling dot product attention model calculation formula is: The scaling dot product attention calculation process is as follows: (1) The normalized input image is represented by X = [X1, X2, ..., XT] to represent T block regions. For each region, three different vectors are obtained by multiplying it with three different linear transformation matrices. Here, the three vectors are named Q, K, and V respectively; Q = WqX, K = WkX, V = WvX, Wq, Wk, and Wv are linear transformation matrices, and d represents the dimension of Q and K. (2) Calculate the attention score, which is the contribution weight of each region to the final classification task. The result is the attention score by performing a dot product operation between the Q matrix and the K matrix of each part. (3) The attention score calculated above is scaled down proportionally to make the model have a more stable gradient, and the result is normalized by Softmax. (4) Multiply the scores by V to obtain a weighted feature information, and add these feature information together to obtain the final output tensor z of the algorithm.
6. The lung cancer pathological image classification method based on the Vision Transformer model according to claim 1, characterized in that: Based on the total number of lung cancer pathology images in each category of the dataset, the number of training iterations for the model was initially determined. The number of different types of lung cancer pathology images in the statistical dataset is counted. Based on the size and complexity of the dataset, the number of training iterations is initially determined. The appropriate batch size is selected based on the performance during training. During training, data augmentation techniques, including rotation, scaling, flipping, and color transformation, are used to increase data diversity.
7. The lung cancer pathological image classification method based on the Vision Transformer model according to claim 1, characterized in that: The preprocessed classification dataset is fed into the model for training. During the model training process, the number of iterations is set to 100, the batch size is 32, and the learning rate is 0.
001. Through multiple iterations and backpropagation, the model parameters are updated to improve the classification accuracy of the model.
8. The lung cancer pathological image classification method based on the Vision Transformer model according to claim 1, characterized in that: Based on the different categories of lung cancer pathology images, construct the hazards and corresponding recommended treatment and rehabilitation measures; Recommended treatment measures: Grade 1: Early-stage lung cancer Predictive characteristics: The tumor is small, confined to the lungs, and has not metastasized to lymph nodes or distant sites; Recommended treatment: The patient is in good health and surgical removal of the tumor is recommended. Surgical types include wedge resection, lobectomy, or pneumonectomy, depending on the location and size of the tumor. For patients with a risk of recurrence after surgery, local radiotherapy is recommended to reduce the risk of recurrence. Regular CT scans and blood tests are required after surgery to monitor for recurrence or metastasis. Grade 2: Locally Advanced Lung Cancer Predictive characteristics: The tumor is large, invading surrounding tissues or having local lymph node metastasis, but no distant metastasis has occurred; Recommended treatment: Chemotherapy should be administered before surgery to shrink the tumor size. The chemotherapy regimen includes platinum-based combination therapy. After neoadjuvant chemotherapy, if the tumor has shrunk and the patient's health permits, surgical resection should be performed. Postoperative chemotherapy or radiotherapy should be administered to eliminate residual cancer cells and reduce the risk of recurrence. Grade 3: Advanced or metastatic lung cancer Predictive characteristics: The tumor has spread to distant organs or other parts of the body, including bones, liver, and brain; Recommended treatment measures: For patients with severe symptoms, systemic chemotherapy is recommended to control tumor progression and alleviate symptoms. Chemotherapy regimens include platinum-based combination therapy and the use of immune checkpoint inhibitors to enhance the patient's immune system's ability to attack cancer cells, including nivolumab and pembrolizumab. For patients with severe and incurable symptoms, palliative care is provided to relieve pain and other symptoms. Palliative care includes analgesics, radiation therapy to relieve symptoms, nutritional support, and psychological counseling.
9. The lung cancer pathological image classification method based on the Vision Transformer model according to claim 1, characterized in that: First, the pathological images of lung cancer to be detected are preprocessed. This involves scaling each image sequentially by reducing the smallest side to 256 pixels and scaling the other side proportionally. Next, each image is cropped from its center along both sides, resulting in a 224×224 image size. Third, the processed images are converted to tensor format using the PyTorch deep learning framework. Fourth, the converted tensor data is standardized channel-by-channel, resulting in a mean and standard deviation of 0.
5. The processed images are then fed into a trained model for prediction. The model sends the prediction results to a database module for data matching. Finally, suggested treatment measures for the corresponding category are extracted from the database module and output along with the detection and classification results.
10. The lung cancer pathological image classification method based on the Vision Transformer model according to claim 1, characterized in that: During model training and use, cross-validation was used to evaluate the model's generalization ability and prevent overfitting. The model's hyperparameters, learning rate, batch size, and number of Transformer layers were adjusted. New lung cancer pathological image data were collected to enrich the training dataset, improve the model's robustness and accuracy, and the model structure and training strategy were improved in a targeted manner.