Ultrasound image assisted diagnosis method for thyroid nodules fusing medical prior knowledge

Abstract

The application discloses a kind of fusion medical prior knowledge's thyroid nodule ultrasonic image auxiliary diagnosis method, comprising the following steps: S1, establishes thyroid nodule ultrasonic image database, obtains original thyroid nodule ultrasonic image specific classification category;S2, original thyroid nodule ultrasonic image is removed to artificial artifact;S3, the region of interest of thyroid nodule ultrasonic image is obtained;S4, the shape, boundary, aspect ratio, echo, calcification feature mentioned in TI-RADS is quantified;S5, the ShuffleNet network of introducing self-attention mechanism is trained, and the overall feature of image is extracted by output layer;S6, medical prior knowledge and depth feature are fused to obtain final feature vector;S7, the feature vector finally obtained is sent into XGBoost classifier to realize the classification of thyroid nodule ultrasonic image.The application solves the classification difficult problem caused by the lack of quantitative index of thyroid nodule.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer-aided diagnosis of thyroid nodules, and in particular to a method for assisting in the diagnosis of thyroid nodule ultrasound images by fusing medical prior knowledge. BACKGROUND

[0002] The incidence of thyroid cancer has shown a rapid upward trend in recent years. Ultrasound, as the preferred method of examination for thyroid lesions, can not only discover lesions but also make a preliminary judgment of their biological behavior, and has the advantages of convenience, safety, etc., but it is somewhat operator-dependent. In the diagnosis of thyroid nodule disease, ultrasound examination in the field of imaging has the advantages of timeliness, convenience, safety, and no radiation, and has become the main screening method for the clinical diagnosis of thyroid nodules. Doctors can make early diagnosis by observing the imaging morphology of thyroid nodules using affordable and harmless ultrasound tools, which will provide important reference value for preoperative disease assessment and subsequent surgical planning. Before artificial intelligence is popularized in the medical field, radiologists mostly observe ultrasound images and describe the thyroid nodule lesion area, risk assessment and classification according to the thyroid imaging reporting and data system (TI-RADS), and under the pressure of a large amount of work, it is inevitable that there will be problems of benign misdiagnosis and malignant misdiagnosis, and the advent of computer-aided diagnosis (CAD) technology makes it possible to solve these problems.

[0003] With the continuous development of deep learning technology, traditional ultrasound diagnosis technology is being combined with cutting-edge artificial intelligence technology to assist doctors in diagnosing benign and malignant thyroid nodules, which has become a trend. The research goals of artificial intelligence combined with thyroid nodule ultrasound diagnosis are mostly focused on two directions: one is the improvement of nodule classification accuracy, and the other is the improvement of nodule lesion positioning and segmentation accuracy. Previously, most research focused on manually extracting morphological and texture features of ultrasound images using image processing techniques, and then using support vector machines (SVM), AdaBoost, and other classifiers to complete simple benign and malignant classification. The advent of convolutional neural networks provides a new approach to the classification of thyroid nodule ultrasound images. However, these methods also have certain shortcomings: the ultrasound images collected in the clinic have poor overall quality, low contrast, and various artifacts due to their inherent imaging mechanism; thyroid nodules vary in shape and size, with fuzzy edges and uneven internal echo distribution; current thyroid nodule grading mainly relies on the experience and observation of doctors, and lacks quantitative index support. This makes the grading of nodules less accurate and repeatable. Therefore, it is difficult for general traditional machine learning or deep learning methods to extract representative features of thyroid nodule ultrasound images. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a thyroid nodule ultrasound image auxiliary diagnosis method fusing medical prior knowledge, which solves the difficulty in grading thyroid nodules due to the lack of quantitative indicators.

[0005] To solve the above technical problems, the technical solution adopted by the present application is as follows: a thyroid nodule ultrasound image auxiliary diagnosis method fusing medical prior knowledge, comprising the following steps:

[0006] Step S1, establishing a thyroid nodule ultrasound image database to obtain the specific classification of the original thyroid nodule ultrasound image;

[0007] Step S2, image preprocessing: removing artificial artifacts from the original thyroid nodule ultrasound image and performing data enhancement;

[0008] Step S3, interested region extraction: using the good segmentation performance of the U-net network to obtain the interested region of the thyroid nodule ultrasound image;

[0009] Step S4, quantification of medical prior knowledge: quantifying the shape, boundary, aspect ratio, echo, and calcification features mentioned in TI-RADS;

[0010] Step S5, extraction of deep features: training a ShuffleNet network with a self-attention mechanism to increase the receptive field of deep networks, compress image information, and extract overall features of the image through the output layer;

[0011] Step S6, feature fusion: fusing the medical prior knowledge and deep features to obtain the final feature vector;

[0012] Step S7, classification: inputting the final feature vector into an XGBoost classifier to realize the grading of the thyroid nodule ultrasound image.

[0013] Further improvement of the technical solution of the present application is that the thyroid nodule classification in step S1 includes 2 classes, 3 classes, 4a class, 4b class, 4c class, and 5 class.

[0014] Further improvement of the technical solution of the present application is that in step S7, the specific operation method of XGBoost classification is as follows: in step S2, the original thyroid nodule ultrasound image is subjected to Roberts operator edge detection to extract a mask image of artificial artifacts, the artifacts are removed through image addition operation, and finally the image is repaired using a fast marching algorithm.

[0015] Further improvement of the technical scheme of the application is that: the specific operation method of XGBoost classification in step S7 is as follows: after the image is repaired by the fast marching algorithm in step S2, the original thyroid nodule ultrasound image is subjected to mixed median filtering to remove image noise, and then contrast-limited adaptive histogram equalization is performed to enhance the contrast of the image with a small dynamic range, so as to improve the accuracy of the classification result.

[0016] Further improvement of the technical scheme of the application is that: the specific operation method of XGBoost classification in step S7 is as follows: in step S4,

[0017] For shape features, circular compactness and ellipticity are selected as shape feature quantities;

[0018] For boundary features, edge point sharpness and edge strength are selected as boundary feature quantities;

[0019] For the aspect ratio, the ratio of the height to the width of the minimum circumscribed rectangle of the nodule is selected to represent;

[0020] For echo features, contrast, entropy, energy, inverse variance, correlation, and variance obtained from the gray level co-occurrence matrix describing the texture features of the image are selected as echo feature quantities;

[0021] For calcification features, the micro-calcification degree is selected as the calcification feature quantity, and the micro-calcification degree refers to the area relationship between the calcification region and the nodule region;

[0022] The above-mentioned total of 12 feature quantities are sequentially spliced to generate a feature vector of medical prior knowledge.

[0023] Further improvement of the technical scheme of the application is that: the specific operation method of XGBoost classification in step S7 is as follows: the specific steps of extracting deep features in step S5 are as follows:

[0024] Step S51, a basic ShuffleNet_V2 model is built, and pre-trained weights on ImageNet are loaded;

[0025] Step S52, a self-attention mechanism is introduced; a multi-head self-attention mechanism is added after each inverted residual module of the ShuffleNet_V2 model, the self-attention layer calculates the similarity between features to obtain more rich information of the image, and ensures that the loss of feature information is minimized;

[0026] Step S53, a Dropout regularization method is selected to prevent overfitting, and features before being input into the fully connected layer are extracted from the last global pooling layer.

[0027] Further improvement of the technical scheme of the present application is that: the specific operation method of XGBoost classification in step S7 is as follows: the early fusion strategy is selected in step S6, and the medical prior knowledge features and deep features are spliced together by concat operation to obtain the fused features.

[0028] Further improvement of the technical scheme of the present application is that: the specific operation method of XGBoost classification in step S7 is as follows: step S71, taking the features obtained by step S6 as the input of the XGBoost classifier;

[0029] Step S72, determining the optimal parameters of the XGBoost classifier: the number of iterations, the minimum sub-weight, the maximum depth of the tree, the minimum loss, the sub-sample ratio of the training instance, the regularization term of the L1 weight, the regularization term of the L2 weight, and the learning rate;

[0030] Step S73, selecting five-fold cross-validation, and using the determined classifier parameters to perform multi-classification on the thyroid nodule ultrasound images.

[0031] Due to the adoption of the above technical scheme, the present application has achieved the following technical progress:

[0032] 1. The medical prior knowledge and deep feature fusion thyroid nodule ultrasound image diagnosis method proposed by the present application can complete the grading of 2, 3, 4a, 4b, and 5 classes of thyroid nodules (1 class and 6 class are determined benign and malignant nodules respectively) in the thyroid imaging reporting and data system (TI-RADS) mentioned in each feature. The features belong to a kind of medical prior knowledge, which is an important basis for doctors to grade thyroid nodules. In order to more accurately grade, the medical prior knowledge is first converted into various feature quantities through feature engineering, and then combined with the deep features of the ultrasound image extracted by the self-attention mechanism improved ShuffleNet network to complete the grading of the thyroid nodule. This method can effectively eliminate artifacts, solve the problem of unclear image noise and nodule area, thereby effectively improving the image quality and improving the accuracy of nodule grading;

[0033] 2. The present application designs an artifact removal algorithm based on the fast marching method (FMM) for artificial artifacts in ultrasound images, combines a hybrid median filter operation to remove image noise, and performs contrast limited adaptive histogram equalization (CLAHE) to enhance the contrast of images with small dynamic range, thereby improving the accuracy of the classification result;

[0034] 3. The ShuffleNet_V2 network is introduced into the self-attention module; the self-attention module calculates the correlation between different feature points in the feature space, fuses the information of different channels, and makes the information contained in the deep feature more abundant; a plurality of self-attention modules constitute a multi-head self-attention mechanism, which is combined with the inverted residual structure of the network structure itself, each inverted residual-self-attention module solves the problems of gradient explosion and gradient disappearance on the one hand, and the extracted channel and global information are more abundant, thereby improving the performance of the whole model;

[0035] 4. The feature based on medical prior knowledge of the present application is the quantification of the feature description of thyroid nodules by TI-RADS, which is more conducive to completing the multi-classification task of thyroid nodules; the deep features extracted based on deep learning can describe the higher dimensional features of thyroid nodules; the mutual fusion of the two kinds of features makes the multi-classification task of thyroid nodules more accurate;

[0036] 5. Compared with the conventional simple benign and malignant classification, the classification is more comprehensive; according to the risk level evaluation mentioned in TI-RADS, 2 classes, 3 classes, 4a classes, 4b classes, 4c classes and 5 classes are selected as the classification results of thyroid nodules, the classification is more accurate, which is helpful for doctors to take timely and targeted measures and timely follow-up treatment surgery for patients. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 It is a flowchart of the method of the present application;

[0038] Figure 2 It is a whole architecture diagram of the present application;

[0039] Figure 3 (a) is an original thyroid nodule ultrasound image;

[0040] Figure 3 (b) is an image obtained by removing artificial artifacts from the original thyroid nodule ultrasound image of the present application;

[0041] Figure 3 (c) is an image obtained by performing hybrid median filtering and contrast limited adaptive histogram equalization (CLAHE) processing on figure 3 (b) of the present application;

[0042] Figure 3 (d) is an image obtained by extracting the region of interest from figure 3 (c) of the present application;

[0043] Figure 4 (a) is a multi-head self-attention module structure diagram of the present application;

[0044] Figure 4 (b) is a single-head self-attention module structure diagram of the present application;

[0045] Figure 5 It is an improved ShuffleNetV2 network model improved by the present application. DETAILED DESCRIPTION

[0046] The application will be further described in detail below in conjunction with examples:

[0047] Example 1

[0048] As Figure 1 shown, a thyroid nodule ultrasound image auxiliary diagnosis method fusing medical prior knowledge includes the following steps:

[0049] Step S1, a thyroid nodule ultrasound image database is established, and the specific classification of the original thyroid nodule ultrasound image is obtained; the original data set is collected from thyroid nodule patients in a hospital, and the grading results of all samples are given by experienced radiologists. The entire data set has a total of 3685 original images, of which 595 belong to class 2, 560 belong to class 3, 625 belong to class 4a, 455 belong to class 4b, 490 belong to class 4c, and 960 belong to class 5. (Class 1 and class 6 are benign and malignant nodules respectively).

[0050] Step S2, image preprocessing: the original thyroid nodule ultrasound image is first subjected to Roberts operator edge detection, the mask image of artificial artifacts is extracted, the artifacts are removed through image addition operation, and finally the image is repaired using fast marching method (FMM). After the repair is completed, a hybrid median filter operation is performed to remove image noise, a contrast limited adaptive histogram equalization (CLAHE) is performed to enhance the contrast of images with small dynamic range, so as to improve the classification result accuracy. As shown in FIG. 3(a), it is an original thyroid nodule image, FIG. 3(b) is an image after removing artificial artifacts and repairing the original thyroid nodule image, and FIG. 3(c) is an image after hybrid median filtering and contrast limited adaptive histogram equalization (CLAHE) processing of the repaired image.

[0051] Step S3, region of interest (ROI) extraction: the U-net network is used to obtain the region of interest of the thyroid nodule ultrasound image due to its good segmentation performance.

[0052] Step S4, quantification of medical prior knowledge: the five features mentioned in TI-RADS, shape, boundary, aspect ratio, echo, and calcification, are quantified. For the shape feature, two indicators, circular compactness and ellipticity, are selected as shape feature quantities;

[0053] For the boundary feature, two indicators, edge point sharpness and edge strength, are selected as boundary feature quantities;

[0054] For the aspect ratio, the ratio of the height to the width of the minimum circumscribed rectangle of the nodule is used to represent the aspect ratio;

[0055] For the echo feature, six statistics of contrast, entropy, energy, inverse variance, correlation and variance obtained from the gray level co-occurrence matrix (GLCM) commonly used to describe the texture features of the image are selected as the echo feature quantity;

[0056] For the calcification feature, the micro-calcification degree is selected as the calcification feature quantity, and the micro-calcification degree refers to the area relationship of the calcification region and the nodule region;

[0057] The above-mentioned total of 12 feature quantities are sequentially spliced to generate the feature vector of the medical prior knowledge.

[0058] The feature based on the medical prior knowledge of the application is the quantification of the TI-RADS feature description of the thyroid nodule, which is more conducive to completing the multi-classification task of the thyroid nodule; the deep feature extracted based on the deep learning can describe the higher-dimensional features of the thyroid nodule; the mutual fusion of the two kinds of features makes the multi-classification task of the thyroid nodule more accurate.

[0059] Step S5, extraction of the deep feature: the ShuffleNet network with the self-attention mechanism is trained to increase the deep network receptive field, the image information is compressed, and the overall features of the image are extracted through the output layer.

[0060] Specifically, the pre-trained weight loaded on ImageNet is introduced into a self-attention mechanism (Self-attention), a multi-head self-attention mechanism is introduced after the last reverse residual module in each stage of a ShuffleNet_V2 model, and a reverse residual-self-attention structure is formed, as shown in FIG. 4(a) and FIG. 4(b). The ShuffleNet_V2 model has two convolutional layers, one maximum pooling layer, one global pooling layer, three stage layers, the number of reverse residual modules in each stage layer is 4, 8 and 4 respectively. The self-attention layer can calculate the similarity between features to obtain more rich information of the image and ensure that the feature information loss is minimized. The reverse residual structure solves the problems of gradient explosion and gradient disappearance and makes the channel direction information extracted more rich. The entire reverse residual-self-attention module can effectively improve the performance of the entire model; finally, the Dropout regularization method is added after each convolutional layer and stage layer to prevent overfitting, and the features before entering the fully connected layer are extracted from the last global pooling layer. The self-attention module is introduced into the ShuffleNet_V2 network; the self-attention module calculates the correlation between different feature points in the feature space, fuses the information of different channels, and makes the information contained in the deep features more rich; the multi-head self-attention mechanism is formed by multiple self-attention modules, combined with the reverse residual structure of the network structure itself, each reverse residual-self-attention module solves the problems of gradient explosion and gradient disappearance on one hand and extracts more rich channel and global information on the other hand, thereby improving the performance of the entire model.

[0061] Step S6, feature fusion: the early fusion strategy is selected to fuse the medical prior knowledge features and the deep features. Specifically, the two parts of features are spliced together through concatenation (Concat) to obtain the fused features.

[0062] Step S7, classification: the last obtained feature vector is sent into an XGBoost classifier to realize the grading of the thyroid nodule ultrasound images. The features obtained in step S6 are taken as the input of the XGBoost classifier, and the best parameters of the XGBoost classifier are determined: the number of iterations, the minimum subweight, the maximum depth of the tree, the minimum loss, the sub-sample ratio of the training instance, the regularization term of the L1 weight, the regularization term of the L2 weight, the learning rate, and the parameter adjustment strategy is selected as the individual parameter adjustment method to avoid the phenomenon of overfitting caused by the high complexity of the model when all parameters are adjusted at the same time; finally, the input is subjected to five-fold cross-validation, and the determined classifier parameters are used to perform multi-classification on the thyroid nodule ultrasound images.

[0063] The application provides a medical prior knowledge and deep feature fusion thyroid nodule ultrasound image diagnosis method, which can complete the grading of 2, 3, 4a, 4b and 5 (1 and 6 are determined benign and malignant nodules respectively) of each feature mentioned in the thyroid imaging reporting and data system (TI-RADS) of the thyroid nodule. The feature belongs to a kind of medical prior knowledge, which is an important basis for doctors to grade the thyroid nodule. In order to more accurately grade, the medical prior knowledge is first converted into various feature quantities through feature engineering, and then combined with the deep features of the ultrasound image extracted by the ShuffleNet network improved by adding the self-attention mechanism to complete the grading of the thyroid nodule. The method can effectively eliminate the artifact, solve the problems of image noise and unclear nodular region, thereby effectively improving the image quality and improving the grading accuracy of the nodule.

Claims

1. A method of ultrasound image assisted diagnosis of thyroid nodules incorporating medical prior knowledge, characterized in that: The method comprises the following steps: Step S1, establishing a thyroid nodule ultrasound image database, obtaining original thyroid nodule ultrasound images, and classifying specific thyroid nodules; Step S2, image preprocessing: removing artificial artifacts from the original thyroid nodule ultrasound images, and performing data enhancement; Step S3, interested region extraction: using the good segmentation performance of the U-net network to obtain the interested region of the thyroid nodule ultrasound image; Step S4, quantification of medical prior knowledge: quantifying the shape, boundary, aspect ratio, echo, and calcification features mentioned in TI-RADS; Step S5, extraction of deep features: training a ShuffleNet network with a self-attention mechanism to increase the receptive field of deep networks, compress image information, and extract overall image features through the output layer; The specific steps of extracting deep features are as follows: Step S51, building a basic ShuffleNet_V2 model and loading pre-trained weights on ImageNet; Step S52, introducing a self-attention mechanism; adding a multi-head self-attention mechanism after each inverted residual module of the ShuffleNet_V2 model, calculating the similarity between features to obtain more rich image information, and ensuring that feature information loss is minimized; Step S53, selecting a Dropout regularization method to prevent overfitting, and extracting features before entering the fully connected layer from the last global pooling layer; Step S6, feature fusion: fusing medical prior knowledge and deep features to obtain a final feature vector; Step S7, classification: inputting the final feature vector into an XGBoost classifier to implement thyroid nodule ultrasound image classification.

2. The method of claim 1, wherein the method further comprises: The thyroid nodule classification categories in step S1 include 2 categories, 3 categories, 4a categories, 4b categories, 4c categories, and 5 categories. 3.The method of claim 1, wherein the method further comprises: In step S2, the original thyroid nodule ultrasound images are subjected to Roberts operator edge detection, the mask image of artificial artifacts is extracted, the artifacts are removed through image addition operation, and finally the image is repaired using the fast marching algorithm.

4. The method of claim 3, wherein the method further comprises: After repairing the image using the fast marching algorithm in step S2, the original thyroid nodule ultrasound images are subjected to hybrid median filtering to remove image noise, and then contrast-limited adaptive histogram equalization is performed to enhance the contrast of images with small dynamic range, thereby improving the classification result accuracy.

5. The method of claim 1, wherein the method further comprises: In step S4: For shape features, circular compactness and ellipticity are selected as shape feature quantities; For boundary features, edge point sharpness and edge intensity are selected as boundary feature quantities; For aspect ratio, the ratio of the height to the width of the minimum bounding rectangle of the nodule is used to represent it; For echo features, contrast, entropy, energy, inverse variance, correlation, and variance obtained from the gray level co-occurrence matrix describing image texture features are used as echo feature quantities; For calcification features, the microcalcification degree is used as a calcification feature quantity, which refers to the area relationship between the calcification region and the nodule region; The shape feature quantities, boundary feature quantities, aspect ratios, echo feature quantities, and calcification feature quantities are sequentially concatenated to generate a feature vector of medical prior knowledge.

6. The method of claim 1, wherein the method further comprises: The early fusion strategy is selected in the step S6, and the medical prior knowledge features and the deep features are spliced by using a concat operation to obtain fused features.

7. The method of claim 6, wherein the method further comprises: The specific operation method of the XGBoost classification in the step S7 is as follows: In step S71, the features obtained in step S6 are used as the input of the XGBoost classifier. In step S72, the optimal parameters of the XGBoost classifier are determined, including the number of iterations, the minimum sub-weight, the maximum depth of the tree, the minimum loss, the sub-sample ratio of the training instance, the regularization term of the L1 weight, the regularization term of the L2 weight, and the learning rate. In step S73, five-fold cross-validation is selected, and the determined classifier parameters are used for multi-classification of the thyroid nodule ultrasound images.

Images