Adenoma and adenocarcinoma ultrasound image classification system and method based on deep learning
By using a deep learning-based multi-scale image processing and feature extraction module, the problem of accurately distinguishing between adenomas and adenocarcinomas in ultrasound images was solved, achieving efficient automated diagnosis, improving classification accuracy, and providing a basis for clinical treatment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI SOUNDWISE TECHNOLOGY CO LTD
- Filing Date
- 2023-05-23
- Publication Date
- 2026-07-14
AI Technical Summary
Current technology makes it difficult to accurately distinguish between follicular thyroid adenoma and adenocarcinoma on ultrasound images, leading to misdiagnosis and missed diagnosis. Furthermore, it relies on physician experience and lacks efficient and automated diagnostic methods.
By employing a deep learning-based multi-scale image processing module, feature extraction module, and classification module, and performing multi-scale transformation and enhancement on ultrasound images, combined with a residual block cascade structure and arbitration algorithm, adenomas and adenocarcinomas can be classified.
It improves the classification accuracy of adenomas and adenocarcinomas, provides a basis for clinical diagnosis, avoids overtreatment, and enhances the reliability and robustness of image features.
Smart Images
Figure CN116704242B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning technology, and in particular to a deep learning-based ultrasound image classification system and method for adenomas and adenocarcinomas. Background Technology
[0002] Follicular thyroid tumors are mainly divided into follicular thyroid adenoma (FTA) and follicular thyroid carcinoma (FTC). FTC is the second most common type of differentiated thyroid cancer, accounting for approximately 10%-20% of all thyroid cancers. It mainly metastasizes to distant sites via hematogenous spread, with the lungs and bones being the most common sites of metastasis. Compared to papillary thyroid carcinoma (PTC), FTC has a lower incidence rate, but a higher rate of distant metastasis and mortality.
[0003] Currently, the main diagnostic methods for follicular thyroid tumors include conventional ultrasound, fine-needle aspiration cytology (FNAC), diagnostic surgical resection, CT, and MRI. Because the cytological characteristics of FTA and FTC are very similar, and FNAC is often limited by the puncture site, making pathological diagnosis difficult, conventional ultrasound examination has advantages such as being non-invasive, radiation-free, multi-angle, and low-cost. Compared with medical imaging methods such as CT and MRI, it is the most commonly used clinical method for examining thyroid diseases. However, FTA and FTC have a lot of overlap on ultrasound images, making it difficult to distinguish them based on size, nodule shape, echogenicity, halo, calcification, blood flow, and other relevant information. Although there is a standard clinical thyroid imaging reporting and data system (TI-RADS), ultrasound physicians with varying experience and skills have a certain degree of subjectivity in interpreting ultrasound images, inevitably leading to misdiagnosis or missed diagnosis. The American Thyroid Association guidelines state that diagnostic surgical resection is the long-established management standard for follicular tumors or suspected tumors, but it may also lead to unnecessary thyroid lobectomy in some adenoma patients, resulting in overtreatment. Therefore, it is necessary to distinguish between FTA and FTC before surgery to provide a basis for clinical patients to choose the best surgical method and avoid overtreatment.
[0004] Due to the uncertainty in the cytological characteristics of FTA and FTC, the differences between them can sometimes be very subtle, making them difficult to distinguish even based on cytological features alone. Furthermore, their morphological appearance on ultrasound images is very similar; their size, shape, and margins can be very similar, making it difficult to determine their type solely through ultrasound imaging. Therefore, accurate diagnosis of FTC and FTA on ultrasound images currently relies primarily on the comprehensive judgment of experienced physicians and pathologists, which is inefficient, highly demanding in terms of experience, and prone to misdiagnosis and missed diagnosis.
[0005] Artificial intelligence (AI) is developing rapidly as an innovative tool and is widely applied in various organ fields of medicine. Compared with traditional ultrasound diagnostic methods, AI can automatically extract image features that are invisible to the naked eye or that ultrasound physicians cannot express verbally using computer algorithms. These image features can be transformed into reliable data information that reflects potential pathophysiology and diagnosis, providing information related to disease diagnosis and prognosis. However, there is currently relatively little research on the differentiation between benign and malignant thyroid follicular tumors.
[0006] In existing technologies, deep learning methods are used to identify ultrasound images of follicular thyroid tumors. By utilizing convolutional neural networks (CNNs) to detect some morphological features of the boundary region of follicular thyroid tumors through image selector sampling and datasets provided by the boundary of follicular thyroid tumors, current methods have low accuracy in distinguishing between FTA and FTC ultrasound images. Summary of the Invention
[0007] To address the above technical problems, this invention provides a deep learning-based ultrasound image classification system and method for adenomas and adenocarcinomas.
[0008] The technical problem solved by this invention can be achieved by the following technical solutions:
[0009] A deep learning-based ultrasound image classification system for adenomas and adenocarcinomas includes:
[0010] The multi-scale image processing module is used to perform multi-scale transformation and enhancement processing on image frames containing ultrasound image data of the target gland to obtain multiple first images at different scales.
[0011] The feature extraction module, connected to the multi-scale image processing module, is used to extract features from multiple first images at different scales to obtain the feature vector corresponding to each first image.
[0012] The classification module, connected to the feature extraction module, is used to classify based on the feature vectors corresponding to the first images at multiple different scales, to obtain the classification result corresponding to the first image at each scale, and to arbitrate the classification results of the first images at all scales to obtain the final classification result.
[0013] Preferably, the multi-scale image processing module includes:
[0014] Multiple image stretching and enhancement units are used to enhance feature images of different scales obtained by decomposing the image frames of ultrasound image data, so as to obtain first images of different scales.
[0015] Preferably, each image stretching and enhancement unit includes:
[0016] The Gaussian filtering subunit is used to perform Gaussian filtering on the feature image to obtain a Gaussian image.
[0017] The first pooling subunit is connected to the Gaussian filtering subunit and is used to perform average downsampling processing on the Gaussian image to obtain the first sampled image.
[0018] The difference processing subunit, connected to the Gaussian filtering subunit, is used to invert the feature image and the Gaussian image to obtain the difference image;
[0019] The second pooling subunit, connected to the difference processing subunit, is used to perform maximum absolute value downsampling on the difference image to obtain the second sampled image;
[0020] An enhancement subunit, connected to a first pooling subunit and a second pooling subunit respectively, is used to add the first sampled image and the second sampled image together to obtain a first image of the corresponding scale.
[0021] Preferably, the feature extraction module includes: multiple residual blocks, which are cascaded and connected to fuse the features of the first image at different scales during downpooling by adding them together to obtain the feature vector corresponding to the first image at each scale, and input the feature vector corresponding to the first image at each scale into the classification module respectively.
[0022] Preferably, the classification module includes:
[0023] Multiple classification units are used to receive feature vectors corresponding to the first image at different scales, and classify them according to the received feature vectors to obtain the classification result corresponding to the first image at each scale.
[0024] The arbitration unit connects to multiple classification units and is used to arbitrate the classification results output by all classification units to obtain the final classification result.
[0025] Preferably, each classification unit includes: a compression and activation network, a flattening layer, a first fully connected layer, a first activation function, a second fully connected layer, and a second activation function connected in series.
[0026] Preferably, the first activation function is a ReLU activation function;
[0027] The second laser function is the Sigmoid activation function.
[0028] Preferably, the arbitration unit includes: a feature concatenation layer and a convolutional neural structure. The input of the feature concatenation layer is connected to multiple classification units, and the output of the feature concatenation layer is connected to the convolutional neural structure. The output of the convolutional neural structure serves as the output of the arbitration unit.
[0029] Preferably, the target gland is the thyroid gland.
[0030] This invention also provides a deep learning-based ultrasound image classification method for adenomas and adenocarcinomas, applied to the deep learning-based ultrasound image classification system for adenomas and adenocarcinomas described above, comprising:
[0031] Multi-scale transformation and enhancement processing are performed on image frames containing ultrasound image data of the target gland to obtain first images at multiple different scales;
[0032] Feature extraction is performed on multiple first images at different scales to obtain the feature vector corresponding to each first image;
[0033] Classification is performed based on the feature vectors corresponding to the first images at multiple different scales to obtain the classification result corresponding to the first image at each scale. The classification results of the first images at all scales are then arbitrated to obtain the final classification result.
[0034] The advantages or beneficial effects of the technical solution of this invention are as follows:
[0035] This invention performs multi-scale transformation and enhancement on ultrasound images, then extracts feature vectors from images at different scales, and performs classification arbitration based on the extracted feature vectors. Based on deep learning, it achieves high classification accuracy in distinguishing ultrasound images of adenomas and adenocarcinomas, providing a basis for clinical patients to select the best surgical method and avoid overtreatment. Attached Figure Description
[0036] Figure 1 A block diagram of a deep learning-based ultrasound image classification system for adenomas and adenocarcinomas is shown in a preferred embodiment of the present invention.
[0037] Figure 2 This is a schematic diagram illustrating a specific implementation of the deep learning-based ultrasound image classification system for adenomas and adenocarcinomas, as a preferred embodiment of the present invention.
[0038] Figure 3 This is a schematic diagram illustrating a specific implementation of the image stretching and enhancement unit in a preferred embodiment of the present invention;
[0039] Figure 4This is a schematic diagram illustrating the specific implementation of the classification unit in a preferred embodiment of the present invention;
[0040] Figure 5 This is a flowchart illustrating a deep learning-based ultrasound image classification method for adenomas and adenocarcinomas, as a preferred embodiment of the present invention. Detailed Implementation
[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0042] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.
[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.
[0044] See Figure 1 and Figure 2 In a preferred embodiment of the present invention, based on the above-mentioned problems existing in the prior art, a deep learning-based ultrasound image classification system for adenomas and adenocarcinomas is provided. This system, based on a multi-scale deep learning network model, distinguishes between FTC and FTA in ultrasound images. The model includes:
[0045] The multi-scale image processing module 1 is used to perform multi-scale transformation and enhancement processing on image frames containing ultrasound image data of the target gland to obtain multiple first images at different scales; wherein, the target gland can preferably be the thyroid gland, and the embodiments of the present invention take follicular thyroid adenoma (FTA) and follicular thyroid carcinoma (FTC) as examples.
[0046] Feature extraction module 2, connected to multi-scale image processing module 1, is used to extract features from multiple first images at different scales to obtain the feature vector corresponding to each first image.
[0047] The classification module 3 is connected to the feature extraction module 2. It is used to classify based on the feature vectors corresponding to the first images at multiple different scales, obtain the classification result corresponding to the first image at each scale, and arbitrate the classification results of the first images at all scales to obtain the final classification result.
[0048] Specifically, in this embodiment, the input ultrasound image is successively scaled down to achieve multi-scale transformation and enhancement, thereby obtaining images of multiple different scales. Then, feature extraction is performed on the scaled-down images to obtain feature vectors of different scales. The feature vectors of each scale are input into a classifier to obtain classification results of different scales. Finally, an arbitration algorithm is used to combine the classification results of multiple scales to obtain the final FTC and FTA classification results. Based on deep learning, the ultrasound images of adenomas and adenocarcinomas are distinguished, effectively enhancing image features and thus improving the accuracy of classification. This provides a basis for clinical patients to select the best surgical method and avoid overtreatment.
[0049] In a preferred embodiment, such as Figure 2 As shown, the multi-scale image processing module 1 includes:
[0050] Multiple image stretching and enhancement units 11 are used to enhance feature images of different scales obtained by decomposing the image frames of ultrasound image data, so as to obtain first images of different scales.
[0051] In a preferred embodiment, such as Figure 3 As shown, each image stretching and enhancement unit 11 includes:
[0052] Gaussian filtering subunit 111 is used to perform Gaussian filtering on the feature image to obtain a Gaussian image;
[0053] The first pooling subunit 113 is connected to the Gaussian filtering subunit 111 and is used to perform average downsampling processing on the Gaussian image to obtain the first sampled image.
[0054] The differential processing subunit 112 is connected to the Gaussian filtering subunit 111 and is used to invert the feature image and the Gaussian image to obtain the differential image.
[0055] The second pooling subunit 114 is connected to the differential processing subunit 112 and is used to perform maximum absolute value downsampling processing on the differential image to obtain the second sampled image;
[0056] The enhancement subunit 115 is connected to the first pooling subunit 113 and the second pooling subunit 114 respectively, and is used to add the first sampled image and the second sampled image to obtain the first image of the corresponding scale.
[0057] Specifically, in this embodiment, an image stretching and enhancement unit 11 is proposed to improve the head layer of the network architecture. A Gaussian pyramid-like structure is adopted to perform multi-scale processing on the input image. For each scale of the ultrasound image, the specific processing method is as follows:
[0058] The input image is Gaussian blurred to obtain a Gaussian image; then the Gaussian image is subtracted from the original input image, and the difference is inverted to obtain a mask difference image, where all pixels with absolute values less than a preset threshold in the difference image are filled with 0; the Gaussian blurred image is downsampled by average, reducing its length and width by half; the difference image is downsampled by maximum absolute value pooling, reducing its length and width by half; finally, the two downsampled images are added together to obtain the first image.
[0059] Compared to traditional multi-scale structures, this invention performs absolute value max pooling on the difference image, effectively preserving key structures and suppressing noise. Simultaneously, the difference image is added back to the Gaussian image, thereby enhancing the structural features of the small-scale image. This is highly relevant to the bubbly structure characteristics of FTC and FTA. Some bubbly structure boundaries are obscured by patch noise, and these boundaries disappear after general absolute value max pooling downsampling. Therefore, boundary enhancement is particularly important in multi-scale image processing. The image obtained through stretching enhancement enhances the structural parts of the image, strengthens image features, and maintains good boundaries, particularly the outer contour of the bubbly structure. This is highly effective in differentiating between benign and malignant bubbly structures.
[0060] The above scheme utilizes multi-scale image processing techniques to decompose and reconstruct the original image at different scales, extracting image features at different scales to achieve a more comprehensive and accurate image analysis and processing method. Multi-scale image processing can decompose the original image to obtain information at different scales, thus better representing features such as edges and textures at different scales. Different decomposition methods are typically used, such as pyramid decomposition and wavelet decomposition. The main advantages of this approach include: multi-scale processing can capture features and details in the image at different scales, thereby improving the reliability and robustness of features; multi-scale image processing can make the algorithm more adaptable to changes at different scales, thus improving its robustness and generalization ability; multi-scale image processing can remove high-frequency components from the original image through decomposition, thereby reducing the impact of noise and improving the accuracy of the algorithm; and multi-scale image processing allows the algorithm to perform calculations at different scales, thereby improving the algorithm's efficiency.
[0061] In a preferred embodiment, such as Figure 2 As shown, the feature extraction module 2 includes: multiple residual blocks 21, which are cascaded and used to fuse the features of the first image at different scales by adding them together during down pooling, so as to obtain the feature vector corresponding to the first image at each scale, and input the feature vector corresponding to the first image at each scale into the classification module respectively.
[0062] In a preferred embodiment, such as Figure 2 As shown, the feature extraction module 2 uses multiple residual blocks 21 to form a stepped multi-layer residual block cascade structure. The number of layers in the residual block cascade structure is the same as the number of image scales, corresponding to the original image frame and the first image processed by the image stretching and enhancement unit 11, respectively. The first residual block of each layer of the residual block cascade structure is used to perform residual processing on the original input image frame or the first image and then input it into the classifier corresponding to the classification module 3. The intermediate residual blocks process the output of the previous residual block of the current layer and input it into the subsequent residual block of the current layer and into the subsequent residual block of the next layer with reduced scale. Then, the output features of the last residual block of all layers are fused using the add method and output to the classifier corresponding to the classification module 3.
[0063] Specifically, by reviewing ultrasound image datasets, it was found that the basic targets of both FTA and FTC lesions are relatively large, requiring a particularly large receptive field for identification. Therefore, high-level feature back-fusion is not very meaningful. In this embodiment, a multi-layer residual cascade approach is proposed to pass features. It borrows the output of each layer of FPN (Feature Pyramid Network). During down-pooling, the features of each layer are fused using the add method, abandoning the upsampling step. Compared with UNET and FPN, it retains the basic features of FTC and FTA, and feeds the results of each layer into inference, reducing the upsampling module and thus reducing the computational load.
[0064] In a preferred embodiment, such as Figure 2 As shown, classification module 3 includes:
[0065] Multiple classification units 31 are used to receive feature vectors corresponding to the first image at different scales, and classify them according to the received feature vectors to obtain the classification result corresponding to the first image at each scale.
[0066] The arbitration unit is connected to multiple classification units 31 and is used to arbitrate the classification results output by all classification units to obtain the final classification result.
[0067] Specifically, in this embodiment, the ultrasound images of adenomas and adenocarcinomas are distinguished by classifying the features of each scale in each layer separately and then arbitrating the classification results of all scale features.
[0068] In a preferred embodiment, each classification unit 31 includes: a compression and activation network (SE), a flattened layer, a first fully connected layer (FC), a first activation function, a second fully connected layer, and a second activation function connected in series.
[0069] In a preferred embodiment, the first activation function is a ReLU activation function;
[0070] The second laser function is the Sigmoid activation function.
[0071] In a preferred embodiment, the arbitration unit includes a feature combination layer 32 and a convolutional neural structure 33. The input of the feature combination layer 32 is connected to multiple classification units 31, and the output of the feature combination layer 32 is connected to the convolutional neural structure 33. The output of the convolutional neural structure 33 serves as the output of the arbitration unit.
[0072] Furthermore, the aforementioned convolutional neural structure 33 includes a convolutional layer and an activation function. The input end of the convolutional layer is connected to the feature union layer 32, and the output end of the convolutional layer is connected to the activation function. The output of the activation function serves as the output end of the convolutional neural structure 33.
[0073] In the preferred embodiment described above, by extracting the largest lesion nodule from the ultrasound image and normalizing the data, the lesion characteristics of the largest preserved nodule in the image are retained before distinguishing between FTA and FTC. Simultaneously, to eliminate the influence of nodule size on classification, a multi-scale cascade operator is introduced, significantly reducing false alarms caused by small nodules.
[0074] This invention also provides a deep learning-based ultrasound image classification method for adenomas and adenocarcinomas, applicable to the aforementioned deep learning-based ultrasound image classification system for adenomas and adenocarcinomas. Figure 5 As shown, it includes:
[0075] S1, perform multi-scale transformation and enhancement processing on the image frame containing the ultrasound image data of the target gland to obtain multiple first images at different scales;
[0076] S2, extract features from multiple first images at different scales to obtain the feature vector corresponding to each first image;
[0077] S3. Classify the first images based on the feature vectors corresponding to multiple different scales to obtain the classification result corresponding to the first image at each scale. Arbitrate the classification results of the first images at all scales to obtain the final classification result.
[0078] Furthermore, the sensitivity and specificity of the model architecture of this invention used to distinguish between FTA and FTC, or the use of existing ResNet-152 and EfficientNet-B7 network models, are shown in Table 1 below:
[0079] Net Sensitivity Specificity ResNet-152 67.2％ 73.3％ EfficentNet-B7 71.5％ 74.9％ This embodiment 78.6％ 77.5％
[0080] Table 1
[0081] The advantages or beneficial effects of the above technical solution are as follows: by sequentially reducing the input ultrasound image, multi-scale transformation and enhancement are achieved to obtain images of multiple different scales; then, feature extraction is performed on the scaled images to obtain feature vectors of different scales; the feature vectors of each scale are input into a classifier to obtain classification results of different scales; then, through an arbitration algorithm, the classification results of multiple scales are combined to obtain the final FTC and FTA classification results. Based on deep learning, the ultrasound images of adenomas and adenocarcinomas are distinguished, effectively enhancing image features and thus improving the accuracy of classification, providing a basis for clinical patients to select the best surgical method and avoid overtreatment.
[0082] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made using the content of this specification and illustrations should be included within the protection scope of the present invention.
Claims
1. A deep learning-based ultrasound image classification system for adenomas and adenocarcinomas, characterized in that, include: The multi-scale image processing module is used to perform multi-scale transformation and enhancement processing on image frames containing ultrasound image data of the target gland to obtain first images at multiple different scales. The feature extraction module, connected to the multi-scale image processing module, is used to extract features from the first images at multiple different scales to obtain the feature vector corresponding to each first image. The classification module, connected to the feature extraction module, is used to classify the first images at multiple different scales based on the feature vectors corresponding to each scale, to obtain the classification result corresponding to each scale of the first image, and to arbitrate the classification results of the first images at all scales to obtain the final classification result. The multi-scale image processing module includes: Multiple image stretching and enhancement units are provided, each of which is used to enhance feature images of different scales obtained by decomposing the image frames of the ultrasound image data to obtain the first image of different scales. Each of the aforementioned image stretching and enhancement units includes: A Gaussian filtering subunit is used to perform Gaussian filtering on the feature image to obtain a Gaussian image; The first pooling subunit is connected to the Gaussian filtering subunit and is used to perform average downsampling processing on the Gaussian image to obtain the first sampled image. The difference processing subunit, connected to the Gaussian filtering subunit, is used to invert the feature image and the Gaussian image to obtain the difference image; The second pooling subunit, connected to the difference processing subunit, is used to perform maximum absolute value downsampling processing on the difference image to obtain the second sampled image; An enhancement subunit, connected to the first pooling subunit and the second pooling subunit respectively, is used to add the first sampled image and the second sampled image to obtain the first image at the corresponding scale.
2. The deep learning-based ultrasound image classification system for adenomas and adenocarcinomas according to claim 1, characterized in that, The feature extraction module includes multiple residual blocks, which are cascaded and used to fuse features of the first image at different scales during downpooling by adding them together to obtain the feature vector corresponding to the first image at each scale. The feature vector corresponding to the first image at each scale is then input into the classification module.
3. The deep learning-based ultrasound image classification system for adenomas and adenocarcinomas according to claim 1, characterized in that, The classification module includes: Multiple classification units are configured to receive the feature vectors corresponding to the first image at different scales, and classify them according to the received feature vectors to obtain the classification result corresponding to the first image at each scale. An arbitration unit, connected to the plurality of classification units, is used to arbitrate the classification results output by all the classification units to obtain the final classification result.
4. The deep learning-based ultrasound image classification system for adenomas and adenocarcinomas according to claim 3, characterized in that, Each of the classification units comprises: a compression and activation network, a flattening layer, a first fully connected layer, a first activation function, a second fully connected layer, and a second activation function, which are connected in series.
5. The deep learning-based ultrasound image classification system for adenomas and adenocarcinomas according to claim 4, characterized in that, The first activation function is a ReLU activation function; The second activation function is the Sigmoid activation function.
6. The deep learning-based ultrasound image classification system for adenomas and adenocarcinomas according to claim 3, characterized in that, The arbitration unit includes a feature concatenation layer and a convolutional neural structure. The input of the feature concatenation layer is connected to the plurality of classification units, and the output of the feature concatenation layer is connected to the convolutional neural structure. The output of the convolutional neural structure serves as the output of the arbitration unit.
7. The deep learning-based ultrasound image classification system for adenomas and adenocarcinomas according to claim 1, characterized in that, The target gland is the thyroid gland.
8. A deep learning-based ultrasound image classification method for adenomas and adenocarcinomas, characterized in that, The system applied to the deep learning-based ultrasound image classification system for adenomas and adenocarcinomas as described in any one of claims 1-7 includes: Multi-scale transformation and enhancement processing are performed on image frames containing ultrasound image data of the target gland to obtain first images at multiple different scales; Feature extraction is performed on the first images at multiple different scales to obtain the feature vector corresponding to each first image; The feature vectors corresponding to the first images at multiple different scales are classified to obtain the classification result corresponding to the first image at each scale. The classification results of the first images at all scales are then arbitrated to obtain the final classification result.