Thyroid nodule ultrasonic image classification system and device based on cooperative training
Through a semi-supervised deep learning method based on collaborative training, the ultrasonic images of thyroid nodules are classified, which solves the problems of insufficient classification accuracy and time-consuming manual labeling data in the prior art, and achieves efficient and accurate classification of thyroid nodules images.
Patent Information
- Application Number
- CN202411684622.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-05-30
AI Technical Summary
The prior art has insufficient accuracy in thyroid nodule ultrasound image classification and requires a large amount of manual labeling data, which is time-consuming and labor-intensive.
Using a semi-supervised deep learning method based on collaborative training, through collaborative training of horizontal and vertical sweep data sets, the classifier after preliminary training is used to classify and predict the unlabeled data, optimize the labeled data set, and retrain until the unlabeled data set is empty.
It significantly improves the accuracy and efficiency of ultrasound image classification of thyroid nodules, reduces manual intervention, and can quickly and accurately judge the properties of thyroid nodules.
Smart Images
Figure CN120070929A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer-aided diagnosis, and particularly to a thyroid nodule ultrasound image classification system and device based on co-training. Background Art
[0002] The statements in this section merely mention the background art related to the present invention and do not necessarily constitute prior art.
[0003] The incidence of thyroid nodules has been increasing year by year. Accurately judging the benign and malignant nature of nodules is an important basis for formulating treatment plans. Ultrasonography is the most widely used method in the diagnosis of benign and malignant nodules, but doctors' diagnoses are highly subjective, and the rates of missed diagnosis and misdiagnosis are relatively high. Although deep neural networks have been widely studied in computer-aided diagnosis (CAD) of different medical images, different ultrasound devices and image patterns pose challenges to clinical applications. Especially in identifying thyroid nodules with various shapes and sizes, the accuracy of identification still fails to meet clinical requirements, and a large amount of manually labeled data is still required, which is time-consuming and laborious. Summary of the Invention
[0004] To solve the deficiencies of the prior art, the present invention provides a thyroid nodule ultrasound image classification system and device based on co-training; the present invention can effectively improve the accuracy and efficiency of thyroid nodule classification with as little manual intervention as possible.
[0005] On the one hand, a thyroid nodule ultrasound image classification system based on co-training is provided, including:
[0006] A training module configured to: train a first classifier by sweeping through the labeled dataset to obtain a preliminarily trained first classifier; train a second classifier by vertically sweeping through the labeled dataset to obtain a preliminarily trained second classifier;
[0007] A prediction module configured to: use the first classifier and the second classifier to classify and predict the horizontally swept unlabeled dataset, merge the images with classification prediction confidence levels higher than a set threshold into the corresponding horizontally swept labeled dataset to obtain an optimized horizontally swept labeled dataset, and delete the merged images from the horizontally swept unlabeled dataset; similarly, obtain an optimized vertically swept labeled dataset, and delete the merged images from the vertically swept unlabeled dataset;
[0008] A retraining module configured to: retrain the corresponding classifiers using the two optimized labeled datasets to obtain a retrained first classifier and a retrained second classifier;
[0009] A repeating module, which is configured to: repeat the working processes of the prediction module and the retraining module until the data in the horizontally scanned unlabeled dataset and the vertically scanned unlabeled dataset are empty, obtaining the finally trained first classifier and the finally trained second classifier; based on the two finally trained classifiers, classify the thyroid nodule image to be classified.
[0010] On the other hand, a thyroid nodule ultrasound image classification device based on co-training is provided, including:
[0011] A thyroid nodule ultrasound image acquisition device, which is used to acquire thyroid nodule ultrasound images and divide the thyroid nodule ultrasound images into a horizontally scanned dataset and a vertically scanned dataset;
[0012] A local storage device, which is used to store the horizontally scanned dataset and the vertically scanned dataset;
[0013] A processor, which is used to divide the horizontally scanned dataset into a horizontally scanned labeled dataset and a horizontally scanned unlabeled dataset; divide the vertically scanned dataset into a vertically scanned labeled dataset and a vertically scanned unlabeled dataset;
[0014] Train the first classifier using the horizontally scanned labeled dataset to obtain a preliminarily trained first classifier; train the second classifier using the vertically scanned labeled dataset to obtain a preliminarily trained second classifier;
[0015] Prediction process: Use the first classifier and the second classifier to classify and predict the horizontally scanned unlabeled dataset, merge the images with classification prediction confidence higher than the set threshold into the corresponding horizontally scanned labeled dataset to obtain an optimized horizontally scanned labeled dataset, and delete the merged images from the horizontally scanned unlabeled dataset; similarly, obtain an optimized vertically scanned labeled dataset, and delete the merged images from the vertically scanned unlabeled dataset;
[0016] Retraining process: Use the two optimized labeled datasets to retrain the corresponding classifiers to obtain the retrained first classifier and second classifier;
[0017] Repeat the working processes of the prediction process and the retraining process until the data in the horizontally scanned unlabeled dataset and the vertically scanned unlabeled dataset are empty, obtaining the finally trained first classifier and the finally trained second classifier; based on the two finally trained classifiers, classify the thyroid nodule image to be classified.
[0018] The above technical solution has the following advantages or beneficial effects:
[0019] This method utilizes computer technology and applies the semi-supervised deep learning method of co-training to the analysis of thyroid nodule ultrasound images. Compared with other methods, this method is fast, has a high accuracy rate, requires only a small number of labeled images, has less manual intervention, can effectively improve the efficiency of thyroid ultrasound image nodule analysis, and can make a quick and accurate judgment on the nature of thyroid nodules.
[0020] The present invention provides a complete co-training-based thyroid nodule ultrasound image stitching and classification system, which provides corresponding deep learning methods from image preprocessing to model training to image classification and realizes each step of the process, fully utilizes the advantages of deep learning methods in image feature extraction and image processing, can effectively improve the accuracy rate of thyroid nodule ultrasound image classification, and can greatly reduce manual intervention, and can better meet clinical needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The accompanying drawings forming a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation of the present invention.
[0022] Figure 1 It is a schematic diagram of the system program module for Embodiment 1. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0023] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.
[0024] Embodiment 1
[0025] This embodiment provides a co-training-based thyroid nodule ultrasound image classification system;
[0026] As Figure 1 shown, the co-training-based thyroid nodule ultrasound image classification system includes:
[0027] An acquisition module, which is configured to: acquire thyroid nodule ultrasound images and divide the thyroid nodule ultrasound images into a transverse scan data set and a longitudinal scan data set;
[0028] A labeling module, which is configured to: divide the transverse scan data set into a transverse scan labeled data set and a transverse scan unlabeled data set; divide the longitudinal scan data set into a longitudinal scan labeled data set and a longitudinal scan unlabeled data set;
[0029] A training module, which is configured to: train a first classifier by sweeping through a labeled dataset to obtain a preliminarily trained first classifier; train a second classifier by vertically sweeping through the labeled dataset to obtain a preliminarily trained second classifier;
[0030] A prediction module, which is configured to: use both the first classifier and the second classifier to perform classification prediction on the unswept unlabeled dataset, merge the images with classification prediction confidence higher than a set threshold into the corresponding swept labeled dataset to obtain an optimized swept labeled dataset, and delete the merged images from the unswept unlabeled dataset; similarly, obtain an optimized vertically swept labeled dataset, and delete the merged images from the vertically swept unlabeled dataset;
[0031] A retraining module, which is configured to: use the two optimized labeled datasets to retrain the corresponding classifiers to obtain a retrained first classifier and a retrained second classifier;
[0032] A repetition module, which is configured to: repeat the working processes of the prediction module and the retraining module until the data in the unswept unlabeled dataset and the vertically swept unlabeled dataset is empty, to obtain a finally trained first classifier and a finally trained second classifier; based on the two finally trained classifiers, classify the thyroid nodule images to be classified.
[0033] Furthermore, there are no duplicate images between the swept dataset and the vertically swept dataset; the swept dataset is a thyroid ultrasound image obtained by a sweeping method; the vertically swept dataset is a thyroid ultrasound image obtained by a vertical sweeping method.
[0034] Exemplarily, the acquisition of thyroid nodule ultrasound images and the division of thyroid nodule ultrasound images into a swept dataset and a vertically swept dataset means that for the collected thyroid ultrasound images, different view classification work is first carried out before benign and malignant classification, and the thyroid ultrasound images of different views obtained by sweeping and vertical sweeping are divided into different datasets.
[0035] Furthermore, the swept labeled dataset refers to an image set obtained by manually labeling the images obtained by the horizontal scanning method for benign or malignant thyroid nodules.
[0036] The vertically swept labeled dataset refers to an image set obtained by manually labeling the images obtained by the vertical scanning method for benign or malignant thyroid nodules.
[0037] Furthermore, an image preprocessing module is also provided between the annotation module and the training module; the image preprocessing module is configured to: perform enhancement processing on the images of both the swept dataset and the vertically swept dataset.
[0038] The image preprocessing module is configured to: perform ROI region extraction operation on the image to extract the key region of the thyroid nodule; perform image filtering processing on the extracted key region to remove the noise in the image; the image filtering processing includes: Gaussian filtering or median filtering.
[0039] It should be understood that after classifying the thyroid ultrasound images of different views, it is also necessary to perform image preprocessing operations to reduce the impact on the results during model training. In the thyroid ultrasound image preprocessing stage, the quality of thyroid ultrasound images from different sources is different, which will greatly affect the subsequent learning of image features. Therefore, it is necessary to perform image preprocessing on the images collected from different views and different ultrasound devices.
[0040] Exemplarily, the preprocessed images of different views are respectively input into two classifiers, which are denoted as the first classifier f1 and the second classifier f2. The models trained on different subsets can be the same or different, as long as they are classifiers that meet the requirements, they can be used as the models of this method for training and testing. The classifier can be implemented by a support vector machine, a random forest, or a convolutional neural network.
[0041] Further, the obtaining of the optimized transverse scanned labeled dataset and the deletion of the merged images from the transverse scanned unlabeled dataset include:
[0042] Using the first classifier and the second classifier, both perform classification prediction on the longitudinal scanned unlabeled dataset, merge the images with classification prediction confidence higher than the set threshold into the corresponding longitudinal scanned labeled dataset to obtain the optimized longitudinal scanned labeled dataset, and delete the merged images from the longitudinal scanned unlabeled dataset.
[0043] Further, the retraining of the corresponding classifier using the two optimized labeled datasets to obtain the retrained classifier includes:
[0044] Using the optimized transverse scanned labeled dataset to retrain the first classifier to obtain the retrained first classifier; using the optimized longitudinal scanned labeled dataset to retrain the second classifier to obtain the retrained second classifier.
[0045] Further, the classification of the thyroid nodule image to be classified based on the two finally trained classifiers specifically includes:
[0046] Based on the finally trained first classifier, the classification of the transverse scanned thyroid nodule image to be classified is realized; based on the finally trained second classifier, the classification of the longitudinal scanned thyroid nodule image to be classified is realized.
[0047] Exemplarily, at the beginning of the training session, the first classifier f1 and the second classifier f2 are first trained according to a given data set. Let the data set be
[0048] S(1) = L(1) ∪ U(1), S(2) = L(2) ∪ U(2);
[0049] where L(1) represents sweeping the labeled data set, L(2) represents vertical sweeping the labeled data set, U(1) represents vertical sweeping the unlabeled data set, and U(2) represents vertical sweeping the unlabeled data set;
[0050] Initially, supervised learning algorithms that sweep the labeled data set L(1) and vertically sweep the labeled data set L(2) are used to learn the first classifier f1 and the second classifier f2 respectively.
[0051] The learning is carried out in an iterative manner. In each iteration, a large number of unlabeled instances are assigned to the pseudo-labels predicted by the first classifier f1 and the second classifier f2, and they are integrated with the swept labeled data set L(1) and the vertically swept labeled data set L(2), so as to retrain and learn the first classifier f1 and the second classifier f2 through the enhanced labeled set.
[0052] In each iteration process, the first classifier f1 and the second classifier f2 predict the labels of each view x(v) ∈ U(v). The top k highest-confidence predictions of the first classifier f1 and the second classifier f2 are used to provide pseudo-labels for the corresponding complementary instances, and then these pseudo-label examples are integrated into the label set of the complementary view.
[0053] In the next iteration, based on the enhanced labeled set, a new model is trained. Even if only one of the views is available to augment the appropriate labeled set, both views of the used instances are removed from U(v).
[0054] Repeat this process until the unlabeled set U(v) is exhausted, generating the corresponding first classifier f1 and second classifier f2. The joint predictions of these models are used to evaluate co-training.
[0055] Furthermore, the first classifier is trained using the swept labeled data set to obtain the preliminarily trained first classifier; the second classifier is trained using the vertically swept labeled data set to obtain the preliminarily trained second classifier. The loss function during the training process is defined as follows:
[0056] Lsup(x,y) = H(y,f1(v1(x)))+H(y,f2(v2(x)))
[0057] Among them, v1(x) and v2(x) are the representations before classification, H(y, f1(v1(x))) and H(y, f2(v2(x))) represent the cross-entropy loss function, v1(x) is the representation of the first classifier f1(v1(x)) before the final classification output layer, v2(x) is the representation of the second classifier f2(v2(x)) before the final classification output layer, and Lsup(x, y) represents the cross-entropy loss function.
[0058] Furthermore, two optimized labeled datasets are used to retrain the corresponding classifiers to obtain the retrained first classifier and second classifier. The JS divergence is used to measure the similarity between f1(v1(x)) and f2(v2(x)). The loss function Lcot(x) is as follows:
[0059]
[0060] The entire training process and model structure reference Figure 1 ; The present invention fully considers the possible problems that may occur during the training of the thyroid nodule ultrasound image classification model. In the ultrasound image preprocessing stage, the ultrasound images collected by different devices are first classified and processed. Since the quality, imaging area, etc. of the ultrasound images collected by different devices are different, preprocessing work needs to be carried out separately. At the same time, device information and patient information should be removed, and manual markings should be removed to obtain preprocessed images.
[0061] During the network training process, the existing thyroid nodule ultrasound images are not sufficient to train two models. Therefore, we use the method of transfer learning and parameter fine-tuning to obtain a model that can be trained with the existing data volume.
[0062] Furthermore, data augmentation is used for the training data, including random flipping and cropping, etc., to expand the data volume of the training data; the increased training image samples after preprocessing and data augmentation can be used to fine-tune the parameters of the first classifier f1 and the second classifier f2 to obtain a deep learning network that can accurately learn and extract thyroid nodule ultrasound images.
[0063] The present invention uses the pytorch framework to fine-tune the above-mentioned first classifier f1 and second classifier f2. First, the training images are changed to processed thyroid nodule images, and then the classification categories of the model are modified to 2, so that the network only learns two output categories corresponding to two cases, namely benign and malignant; after inputting data of different views into different models, the two models first independently train each model on their respective data subsets.
[0064] After each is trained, the two models respectively predict the unlabeled data of different views, select the image with the highest confidence as the pseudo-label and add it to the training set, and continue training. At the same time, the two models perform further training iterations to refine and improve the model performance; repeat until all unlabeled data is correctly classified.
[0065] After training, perform a performance evaluation test on the model, adjust the training strategy or model parameters according to the test results; after the evaluation is completed, deploy the model to the classification system, monitor its performance, and make timely adjustments as needed.
[0066] Embodiment 2
[0067] A thyroid nodule ultrasound image classification device based on co-training includes:
[0068] A thyroid nodule ultrasound image acquisition device for acquiring thyroid nodule ultrasound images and dividing the thyroid nodule ultrasound images into a transverse scan data set and a longitudinal scan data set;
[0069] A local storage device for storing the transverse scan data set and the longitudinal scan data set;
[0070] A processor for dividing the transverse scan data set into a transverse scan labeled data set and a transverse scan unlabeled data set; dividing the longitudinal scan data set into a longitudinal scan labeled data set and a longitudinal scan unlabeled data set;
[0071] Use the transverse scan labeled data set to train the first classifier to obtain the initially trained first classifier; use the longitudinal scan labeled data set to train the second classifier to obtain the initially trained second classifier;
[0072] Prediction process: Use the first classifier and the second classifier to perform classification predictions on the transverse scan unlabeled data set, merge the images with classification prediction confidence higher than the set threshold into the corresponding transverse scan labeled data set to obtain the optimized transverse scan labeled data set, and delete the merged images from the transverse scan unlabeled data set; similarly, obtain the optimized longitudinal scan labeled data set, and delete the merged images from the longitudinal scan unlabeled data set;
[0073] Retraining process: Use the two optimized labeled data sets to retrain the corresponding classifiers to obtain the retrained first classifier and second classifier;
[0074] Repeat the working processes of the prediction process and the retraining process until the data in the transverse scan unlabeled data set and the longitudinal scan unlabeled data set is empty, to obtain the finally trained first classifier and the finally trained second classifier; based on the finally trained two classifiers, classify the thyroid nodule images to be classified.
[0075] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.
Claims
1. A thyroid nodule ultrasound image classification system based on collaborative training, characterized by: include: A training module is configured to: train the first classifier by scanning the labeled data set to obtain the first classifier after preliminary training; The second classifier is trained using the vertical scan labeled data set to obtain a second classifier after preliminary training; The prediction module is configured to: use the first classifier and the second classifier to perform classification prediction on the horizontal sweep unlabeled data set, merge images with classification prediction confidence higher than a set threshold into the corresponding horizontal sweep labeled data set to obtain an optimized horizontal sweep labeled data set, and delete the merged images from the horizontal sweep unlabeled data set; Similarly, the optimized vertical scan labeled data set is obtained, and the merged images are deleted from the vertical scan unlabeled data set; A retraining module is configured to: use the two optimized labeled data sets to retrain the corresponding classifiers to obtain a retrained first classifier and a second classifier; A repetition module is configured to: repeat the working process of the prediction module and the retraining module until the data of the horizontal scanning unlabeled data set and the vertical scanning unlabeled data set are empty, and obtain the final trained first classifier and the final trained second classifier; Based on the two finally trained classifiers, the thyroid nodule images to be classified are classified.
2. The thyroid nodule ultrasound image classification system based on collaborative training as claimed in claim 1, characterized in that: The training module also includes: An acquisition module is configured to: acquire an ultrasonic image of a thyroid nodule, and divide the ultrasonic image of the thyroid nodule into a horizontal scanning data set and a vertical scanning data set; The labeling module is configured to: divide the horizontal scanning data set into a horizontal scanning labeled data set and a horizontal scanning unlabeled data set; and divide the vertical scanning data set into a vertical scanning labeled data set and a vertical scanning unlabeled data set.
3. The thyroid nodule ultrasound image classification system based on collaborative training as claimed in claim 2, characterized in that: There are no repeated images between the horizontal scanning data set and the vertical scanning data set; the horizontal scanning data set is a thyroid ultrasound image obtained by horizontal scanning; and the vertical scanning data set is a thyroid ultrasound image obtained by vertical scanning.
4. The thyroid nodule ultrasound image classification system based on collaborative training as claimed in claim 1, characterized in that: The horizontal scan labeled data set refers to a set of images obtained by manually marking benign or malignant thyroid nodules on images obtained by horizontal scanning; the vertical scan labeled data set refers to a set of images obtained by manually marking benign or malignant thyroid nodules on images obtained by vertical scanning.
5. The thyroid nodule ultrasound image classification system based on collaborative training as claimed in claim 2, characterized in that: An image preprocessing module is also provided between the labeling module and the training module; the image preprocessing module is configured to: perform enhancement processing on the images of the horizontal scanning data set and the vertical scanning data set; The image preprocessing module is configured to: perform ROI region extraction operation on the image to extract the key area of the thyroid nodule; Perform image filtering on the extracted key areas to remove noise from the image; The image filtering process includes: Gaussian filtering or median filtering.
6. The thyroid nodule ultrasound image classification system based on collaborative training as claimed in claim 1, characterized in that: The step of obtaining the optimized horizontally scanned labeled data set and deleting the merged image from the horizontally scanned unlabeled data set includes: The first classifier and the second classifier are both used to perform classification prediction on the vertical scan unlabeled dataset, and the images whose classification prediction confidence is higher than the set threshold are merged into the corresponding vertical scan labeled dataset to obtain the optimized vertical scan labeled dataset, and the merged images are deleted from the vertical scan unlabeled dataset.
7. The thyroid nodule ultrasound image classification system based on collaborative training as claimed in claim 1, characterized in that: The method of using the two optimized labeled data sets to retrain the corresponding classifiers to obtain the retrained first classifier and the second classifier includes: The optimized horizontal scan labeled data set is used to retrain the first classifier to obtain a retrained first classifier; the optimized vertical scan labeled data set is used to retrain the second classifier to obtain a retrained second classifier.
8. The thyroid nodule ultrasound image classification system based on collaborative training as claimed in claim 1, characterized in that: The method of classifying the thyroid nodule images to be classified based on the two finally trained classifiers specifically includes: classifying the horizontally scanned thyroid nodule images to be classified based on the finally trained first classifier; and classifying the vertically scanned thyroid nodule images to be classified based on the finally trained second classifier.
9. The thyroid nodule ultrasound image classification system based on collaborative training as claimed in claim 1, characterized in that: The first classifier is trained by using the horizontal scan labeled data set to obtain the first classifier after preliminary training; the second classifier is trained by using the vertical scan labeled data set to obtain the second classifier after preliminary training. The loss function in the training process is defined as follows: Lsup(x,y)=H(y,f1(v1(x)))+H(y,f2(v2(x))) Among them, v1(x) and v2(x) are the representations before classification, H(y,f1(v1(x))) and H(y,f2(v2(x))) represent the cross entropy loss function, v1(x) is the representation of the first classifier f1(v1(x)) before the final classification output layer, v2(x) is the representation of the second classifier f2(v2(x)) before the final classification output layer, and Lsup(x,y) represents the cross entropy loss function.
10. A thyroid nodule ultrasound image classification device based on collaborative training, characterized in that: include: A thyroid nodule ultrasound image acquisition device is used to acquire a thyroid nodule ultrasound image and divide the thyroid nodule ultrasound image into a horizontal scanning data set and a vertical scanning data set; A local storage device, used for storing the horizontal scanning data set and the vertical scanning data set; A processor, configured to divide the sweeping data set into a sweeping labeled data set and a sweeping unlabeled data set; The vertical scanning dataset is divided into a vertical scanning labeled dataset and a vertical scanning unlabeled dataset; The first classifier is trained by scanning the labeled data set to obtain the first classifier after preliminary training; The second classifier is trained using the vertical scan labeled data set to obtain a second classifier after preliminary training; Prediction process: The first classifier and the second classifier are used to perform classification prediction on the sweeping unlabeled dataset, and the images with classification prediction confidence higher than the set threshold are merged into the corresponding sweeping labeled dataset to obtain the optimized sweeping labeled dataset, and the merged images are deleted from the sweeping unlabeled dataset; Similarly, the optimized vertical scan labeled data set is obtained, and the merged images are deleted from the vertical scan unlabeled data set; Retraining process: using the two optimized labeled data sets, retraining the corresponding classifiers to obtain the retrained first classifier and the second classifier; Repeat the prediction process and the retraining process until the data of the horizontally scanned unlabeled data set and the vertically scanned unlabeled data set are empty, and obtain the final trained first classifier and the final trained second classifier; Based on the two finally trained classifiers, the thyroid nodule images to be classified are classified.