Deep learning-based salivary gland tumor ultrasonic image classification method
Through the ResNet50d model and Focal Loss function based on deep learning, the ultrasound images of salivary glands were classified, which solved the problem of diagnostic uncertainty caused by overlapping features in ultrasound diagnosis, and improved the diagnostic efficiency and accuracy of parotid, submandibular and sublingual salivary gland tumors. , enhances the credibility of the diagnosis.
Patent Information
- Application Number
- CN202410998841.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-12-17
AI Technical Summary
In the prior art, when diagnosing salivary gland tumors ultrasonically, overlapping features leads to uncertain diagnosis, and there is a lack of effective methods to improve the specificity and efficiency of diagnosis, especially when malignant tumors are rare and imaging data are insufficient.
Using a deep learning-based method, the ResNet50d model combined with Focal Loss function is used to classify salivary gland ultrasound images, and the accuracy and credibility of classification are improved through data preprocessing, iterative training and evaluation index optimization.
It improves the diagnostic efficiency and accuracy of parotid, submandibular and sublingual salivary gland tumors, reduces the workload of doctors, enhances the credibility of diagnosis, and improves the prognosis of patients.
Smart Images
Figure CN119152248A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image classification, and in particular to a salivary gland tumor ultrasound image classification method based on deep learning. Background Art
[0002] Salivary gland tumors (SGTs) are challenging to diagnose and treat due to their diverse incidence and clinical behavior. Parotid gland (PG) tumors are the most common, accounting for approximately 80% of major salivary gland tumors, and most are benign. The incidence of malignant tumors in the submandibular gland (SMG) and sublingual gland (SLG) is relatively high, at 41%-45% and 70%-90%, respectively. Surgical resection is the main treatment method. Benign parotid tumors are usually treated with partial resection, while malignant tumors require complete gland resection to ensure a safe margin and reduce recurrence. Accurate preoperative diagnosis is crucial for the formulation of surgical plans and prognosis. Preoperative ultrasound, as an accurate, convenient, and non-invasive technique, is widely used in the diagnosis of salivary gland tumors. Benign tumors usually have clear boundaries, while malignant tumors appear irregular and highly vascularized. However, sometimes overlapping features lead to diagnostic uncertainty, which requires reliance on physician experience. Therefore, new methods are needed to improve the specificity and efficiency of ultrasound diagnosis.
[0003] In recent years, deep learning (DL) models have been widely used in medical imaging, and convolutional neural networks (CNNs) are the most commonly used in medical image processing because they can extract imaging features. DL models process large-scale medical image data and can identify richer information than the human eye, such as the intensity, shape, size, and texture of tumors. However, there are few studies on the application of DL models in ultrasound diagnosis of salivary gland tumors due to the short history of ultrasound diagnosis, the rarity of malignant tumors, and insufficient imaging data. Summary of the invention
[0004] In response to the above problems, the present invention provides a salivary gland tumor ultrasound image classification method based on deep learning, which can effectively improve the diagnostic efficiency in distinguishing benign and malignant salivary gland tumors.
[0005] The technical solution of the present invention is as follows:
[0006] A deep learning-based salivary gland tumor ultrasound image classification method comprises the following steps:
[0007] S1. Annotate the salivary gland ultrasound images of the cases with pathological diagnosis as benign or malignant to obtain annotated salivary gland ultrasound images;
[0008] S2, performing data format conversion, data desensitization, data division and data enhancement preprocessing on the annotated salivary gland ultrasound image to obtain a preprocessed image;
[0009] S3, using the ResNet50d model as the basic model and the Focal Loss function as the loss function;
[0010] S4. Perform deep learning training on the ResNet50d model through repeated iterations, adjust parameters, and output a model whose evaluation results meet the preset requirements;
[0011] S5. Based on the trained ResNet50d model, input the salivary gland ultrasound image to be identified and complete the identification based on the output result.
[0012] In a further technical solution, in step S1, the salivary gland ultrasound images of the cases with pathological diagnosis are stored in JPG format and then labeled as benign or malignant.
[0013] In a further technical solution, in step S2, preprocessing the annotated salivary gland ultrasound image specifically includes the following steps:
[0014] S21, performing size unification processing on the annotated salivary gland ultrasound images to obtain images of uniform size;
[0015] S22, retaining the salivary gland ultrasound image portion by cropping, and deleting the annotation information outside the image and the data source information;
[0016] S23, dividing the data into training set and validation set in a ratio of 6:4, using a 5-fold cross-validation method, and ensuring that all images of the same patient are in the same fold data set when dividing the data;
[0017] S24. Perform data enhancement on all salivary gland ultrasound image data by rotating them by 90°, 180°, and 270° and flipping them horizontally.
[0018] In a further technical solution, in step S3, the ResNet50d model is composed of a stack of four types of residual blocks (Res i dual Blocks) with different numbers of channels, including a total of 49 convolutional layers and one fully connected layer.
[0019] In a further technical solution, in step S4, repeatedly iterating deep learning training of the ResNet50d model specifically includes the following steps:
[0020] S41, initialize the ResNet50d model using weights pre-trained on the ImageNet dataset;
[0021] S42. The initialized ResNet50d model is trained in the Pytorch framework using the AdamW optimizer with a learning rate of 0.001, a weight decay of 1e-4, a batch size of 32, and a default number of training iterations of 200.
[0022] In a further technical solution, in step S4, all iterative deep learning training processes for the ResNet50d model are performed using a single NVID IA GeForce RTX3090 GPU.
[0023] In a further technical solution, in step S4, evaluation indicators including Accuracy, Sensitivity, Specificity, PPV, NPV, F1-Score, Recall and AUC are used to perform task evaluation, and the threshold of the model is adjusted to obtain a model that meets the preset requirements.
[0024] In a further technical solution, in step S4, when the ResNet50d model is subjected to deep learning training through repeated iterations, a random undersampling method is used to extract salivary gland ultrasound image data to reduce the number of benign samples so that the ratio of benign and malignant samples tends to be balanced.
[0025] In a further technical solution, the salivary gland tumor ultrasound image includes a parotid gland tumor ultrasound image, a submandibular gland tumor ultrasound image and a sublingual gland tumor ultrasound image.
[0026] The beneficial effects of the present invention are:
[0027] 1. The present invention provides a salivary gland tumor ultrasound image classification method based on deep learning, which is applicable to parotid gland, submandibular gland, and sublingual gland tumors. The ResNet50d model in deep learning is used as the basic model, and the Focal Loss loss function is used in the model. The network structure is improved to adapt to the benign and malignant classification task of salivary gland tumor ultrasound images, thereby improving the classification accuracy.
[0028] 2. The present invention makes adaptive adjustments to the model based on salivary gland ultrasound images, and uses evaluation indicators including Accuracy, Sensitivity, Specificity, PPV, NPV, F1-Score, Recall and AUC for task evaluation, adjusts the threshold of the model, and obtains the model with the best evaluation results. This model can help doctors make rapid and accurate diagnosis of salivary gland tumors, reduce the workload of doctors, improve the credibility of diagnosis, and is expected to improve patient prognosis. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 is a flow chart of a salivary gland tumor ultrasound image classification method based on deep learning according to an embodiment of the present invention;
[0030] Figure 2It is a schematic diagram of the structure of the ResNet50d model described in an embodiment of the present invention;
[0031] Figure 3 is a comparison chart of the final model DeepSGT described in the embodiment of the present invention, the original ResNet50d model, and the diagnosis results of the ultrasound doctor;
[0032] Figure 4 2 is a confusion matrix diagram of the final model DeepSGT described in the embodiment of the present invention. DETAILED DESCRIPTION
[0033] The embodiments of the present invention will be further described below in conjunction with the accompanying drawings.
[0034] Example:
[0035] like Figure 1 As shown, a deep learning-based salivary gland tumor ultrasound image classification method comprises the following steps:
[0036] S1. Annotate the salivary gland ultrasound images of the cases with pathological diagnosis as benign and malignant, and obtain annotated salivary gland ultrasound images, a total of 315 images, including 217 benign tumor images and 98 malignant tumor images;
[0037] S2, performing data format conversion, data desensitization, data division and data enhancement preprocessing on the annotated salivary gland ultrasound image to obtain a preprocessed image;
[0038] S3. Use the ResNet50d model as the basic model and the Focal Loss function as the loss function. The formula of the Focal Loss function is as follows:
[0039] FocalLoss(pt)=-α(1-pt)γlog(pt)
[0040] Among them, p t is the probability that a sample is correctly classified, αt is the balancing factor, and γ is the adjustment factor. By adjusting the values of α and γ, the focus on difficult-to-classify samples can be controlled, thereby effectively alleviating the impact of category imbalance;
[0041] S4. Train the ResNet50d model through deep learning through repeated iterations, adjust the parameters, and output the model with the best evaluation results;
[0042] S5. Based on the trained ResNet50d model, input the salivary gland ultrasound image to be identified and complete the identification based on the output result.
[0043] In this embodiment, the salivary gland tumor ultrasound image includes a parotid gland tumor ultrasound image, a submandibular gland tumor ultrasound image, and a sublingual gland tumor ultrasound image.
[0044] In this embodiment, the ultrasound image of the patient's salivary gland tumor is obtained by the hospital's ultrasound examination instrument, processed by the instrument's own software, and reviewed by two ultrasound physicians to intercept the cross-section with the largest diameter to obtain image data in JPG format. In addition to the image data, the initial data also includes the patient's pathological diagnosis report, which is the gold standard for determining benign or malignant.
[0045] In this embodiment, in step S2, preprocessing the annotated salivary gland ultrasound image specifically includes the following steps:
[0046] S21, performing size unification processing on the annotated salivary gland ultrasound images to obtain images of uniform size;
[0047] S22, retaining the salivary gland ultrasound image portion by cropping, and deleting the annotation information outside the image and the data source information;
[0048] S23, dividing the data into training set and validation set in a ratio of 6:4, using a 5-fold cross-validation method, and ensuring that all images of the same patient are in the same fold data set when dividing the data;
[0049] S24. Perform data enhancement on all salivary gland ultrasound image data by rotating them by 90°, 180°, and 270° and flipping them horizontally.
[0050] In this embodiment, in step S4, repeatedly iterating deep learning training of the ResNet50d model specifically includes the following steps:
[0051] S41, initialize the ResNet50d model using weights pre-trained on the ImageNet dataset;
[0052] S42. The initialized ResNet50d model is trained in the Pytorch framework using the AdamW optimizer with a learning rate of 0.001, a weight decay of 1e-4, a batch size of 32, and a default number of training iterations of 200.
[0053] The structure of the ResNet50d model is as follows Figure 2As shown, in this embodiment, the weight-initialized ResNet50d model pre-trained on the ImageNet dataset is used, and the Pytorch framework is used to build and train the model. The model is trained using the AdamW optimizer, with a learning rate of 0.001, a weight decay of 1e-4, a batch size of 32, and a default number of training iterations of 200. All experiments are performed using a single NVIDIA GeForce RTX3090 GPU.
[0054] In step S4 of this embodiment, the hardware used for training and subsequent processes is NVID IA GeForceRTX3090 CPU, and all programming is done in Python 3.9.16 and PyTorch 2.0.0 environment. The optimizer used in this embodiment is AdamW, the learning rate is 0.001, the weight decay is 1e-4, the batch size is 32, and the default number of training iterations is 200. Through multiple experiments, the hyperparameters are continuously adjusted to find the best combination and complete 5-fold cross validation.
[0055] In one embodiment, in step S1, the salivary gland ultrasound images of the cases with pathological diagnosis are stored in JPG format and then labeled as benign or malignant.
[0056] In one of the embodiments, in step S3, the ResNet50d model is composed of four types of residual blocks (Res i dual Blocks) stacked with different numbers of channels, including a total of 49 convolutional layers and one fully connected layer.
[0057] In one embodiment, in step S4, all iterative deep learning training processes for the ResNet50d model are performed using a single NVID IA GeForce RTX3090 GPU.
[0058] In one of the embodiments, in step S4, when the ResNet50d model is repeatedly iterated to perform deep learning training, a random undersampling method is used to extract salivary gland ultrasound image data to reduce the number of benign samples so that the ratio of benign and malignant samples tends to be balanced.
[0059] In another embodiment, in step S4, evaluation indicators including Accuracy, Sensitivity, Specificity, PPV, NPV, F1-Score, Recall and AUC are used to perform task evaluation, and the threshold of the model is adjusted to obtain a model that meets the preset requirements.
[0060] The diagnostic efficiency indicators of the final model obtained in this embodiment are shown in the following table. The optimized final model DeepSGT is compared with the original ResNet50d model and the diagnosis results of ultrasound doctors. The results are as follows: Figure 3 and Figure 4 shown.
[0061]
[0062] This embodiment classifies salivary gland tumor ultrasound images based on the ResNet50d model, improves the accuracy of classification, and uses the Focal Loss loss function in the model to adapt to samples with small feature differences. In addition, the model is adaptively adjusted for salivary gland tumor ultrasound images, and the trained model can help doctors make quick and accurate diagnoses, and has certain clinical use potential.
[0063] The above-mentioned embodiments only express the specific implementation of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A deep learning-based salivary gland tumor ultrasound image classification method, characterized in that: The following steps are involved: S1. Annotate the salivary gland ultrasound images of the cases with pathological diagnosis as benign or malignant to obtain annotated salivary gland ultrasound images; S2, performing data format conversion, data desensitization, data division and data enhancement preprocessing on the annotated salivary gland ultrasound image to obtain a preprocessed image; S3, using the ResNet50d model as the basic model and the Focal Loss function as the loss function; S4. Perform deep learning training on the ResNet50d model through repeated iterations, adjust parameters, and output a model whose evaluation results meet the preset requirements; S5. Based on the trained ResNet50d model, input the salivary gland ultrasound image to be identified and complete the identification based on the output result.
2. The deep learning-based salivary gland tumor ultrasound image classification method according to claim 1, characterized in that: In step S1, the salivary gland ultrasound images of the cases that have been pathologically confirmed are stored in JPG format and then labeled as benign or malignant.
3. The deep learning-based salivary gland tumor ultrasound image classification method according to claim 1, characterized in that: In step S2, preprocessing the annotated salivary gland ultrasound image specifically includes the following steps: S21, performing size unification processing on the annotated salivary gland ultrasound images to obtain images of uniform size; S22, retaining the salivary gland ultrasound image portion by cropping, and deleting the annotation information outside the image and the data source information; S23, dividing the data into training set and validation set in a ratio of 6:4, using a 5-fold cross-validation method, and ensuring that all images of the same patient are in the same fold data set when dividing the data; S24. Perform data enhancement on all salivary gland ultrasound image data by rotating them by 90°, 180°, and 270° and flipping them horizontally.
4. The deep learning-based salivary gland tumor ultrasound image classification method according to claim 1, characterized in that: In step S3, the ResNet50d model is composed of four types of residual blocks with different numbers of channels, including 49 convolutional layers and one fully connected layer.
5. The deep learning-based salivary gland tumor ultrasound image classification method according to claim 1, characterized in that: In step S4, the iterative deep learning training of the ResNet50d model specifically includes the following steps: S41, initialize the ResNet50d model using weights pre-trained on the ImageNet dataset; S42. The initialized ResNet50d model is trained in the Pytorch framework using the AdamW optimizer with a learning rate of 0.001, a weight decay of 1e-4, a batch size of 32, and a default number of training iterations of 200.
6. The deep learning-based salivary gland tumor ultrasound image classification method according to claim 5, characterized in that: In step S4, all iterative deep learning training processes for the ResNet50d model are performed using a single NVIDIA GeForce RTX3090 GPU.
7. The deep learning-based salivary gland tumor ultrasound image classification method according to claim 1, characterized in that: In step S4, evaluation indicators including Accuracy, Sensitivity, Specificity, PPV, NPV, F1-Score, Recal and AUC are used to perform task evaluation, and the threshold of the model is adjusted to obtain a model that meets the preset requirements.
8. The deep learning-based salivary gland tumor ultrasound image classification method according to claim 1, characterized in that: In step S4, when the ResNet50d model is repeatedly iterated to perform deep learning training, a random undersampling method is used to extract salivary gland ultrasound image data to reduce the number of benign samples so that the ratio of benign and malignant samples tends to be balanced.
9. The deep learning-based salivary gland tumor ultrasound image classification method according to any one of claims 1 to 8, characterized in that: The salivary gland tumor ultrasound images include parotid gland tumor ultrasound images, submandibular gland tumor ultrasound images and sublingual gland tumor ultrasound images.