Thyroid nodule intelligent segmentation and automatic rating method based on ultrasonic image

Abstract

The invention discloses a thyroid nodule intelligent segmentation and automatic rating method based on an ultrasonic image, and relates to an image intelligent segmentation and automatic rating method. The invention aims to solve the problem that the existing thyroid nodule ultrasonic image cannot realize intelligent segmentation and automatic rating. The method comprises the following steps: segmenting an ultrasonic image by using a provided thyroid nodule segmentation method based on a GADRU-Net network; performing preprocessing operation on the segmented nodule image; 6 diagnostic features are extracted from the nodule according to a rating index specified by C-TIRADS; and classifying by using a support vector machine or a convolutional network, and rating according to indexes. The invention belongs to the technical field of ultrasonic medical image processing.

Description

Technical Field

[0001] The invention relates to an image intelligent segmentation and automatic rating method, belonging to the technical field of ultrasonic medical image processing. Background Art

[0002] In the diagnosis of clinical thyroid diseases, ultrasound imaging technology is often used as an initial detection method. Thyroid nodules are the most common disease detected, and the incidence rate is on the rise. Therefore, thyroid gland health assessment has become the main focus of medical ultrasound diagnosis. Ultrasound imaging technology has the advantages of being non-invasive, radiation-free, and cost-effective. Specialists often obtain a large amount of indirect diagnostic information based on the thyroid nodule area in ultrasound images and their clinical experience, and rate thyroid nodules to provide patients with the tendency of benign and malignant nodules. Therefore, intelligent segmentation of thyroid nodules and rating of thyroid nodules based on the segmentation results are of great clinical application value.

[0003] Traditional thyroid nodule segmentation algorithms mainly include support vector machine and k-nearest neighbor classification algorithm, watershed algorithm, fuzzy C-means clustering algorithm, improved local grayscale clustering method, etc.; in the research of thyroid nodule intelligent segmentation based on deep learning, full convolutional neural network, DeepLab v3+ model, and U-Net are often improved; common traditional methods for automatic rating include: extracting GLCM features and using support vector machine for classification, using extreme learning machine to learn features screened by graph clustering ant colony optimization algorithm, using multi-layer perceptron to learn extracted texture features, and improving Boosting algorithm; common methods for automatic rating of thyroid nodules based on deep learning in recent years include: extracting features from GoogLeNet and VGG network and combining them with traditional features for majority voting mechanism, using transfer learning and support vector machine to integrate multi-depth model outputs, etc.

[0004] However, compared with images of natural objects and the environment, medical ultrasound images are more complex and difficult to distinguish clearly, resulting in low accuracy of segmentation and automatic rating based on raw ultrasound images. In particular, since the acquisition and annotation of medical images rely heavily on the experience and heavy workload of specialists, small sample problems are often encountered in the field of intelligent medical image processing, which also hinders the development of intelligent segmentation and automatic rating methods for thyroid ultrasound images. As for the rating of thyroid nodules, many existing rating methods have incomplete rating standards, only classifying benign and malignant without quantifying various indicators, or are based on deep learning algorithms and lack interpretability. The information obtained is not comprehensive enough to explain its feasibility to doctors. Summary of the invention

[0005] The present invention aims to solve the problem that existing thyroid nodule ultrasound images cannot realize intelligent segmentation and automatic rating, and further proposes a thyroid nodule intelligent segmentation and automatic rating method based on ultrasound images.

[0006] The technical solution adopted by the present invention to solve the above-mentioned problem is: the steps of the present invention include:

[0007] Step 1: Using the proposed GADRU-Net network-based thyroid nodule segmentation method, according to the thyroid ultrasound images of the training set I T and the corresponding segmentation labels Train the network and get Among them, GR is the proposed GADRU-Net segmentation network, M DR 、M LM and M TF They are the dual residual multi-convolution module, texture modality feature extraction module and supplementary connection module in the segmentation network; then the test set image I V Perform segmentation and obtain segmented nodule labels

[0008] Step 2: Use the nodule labels obtained by segmentation to extract the nodule part corresponding to the original ultrasound image, and perform preprocessing operations, including gray value mapping, convex hull extraction, nodule internal area extraction, nodule boundary area extraction, and extraction of the neighborhood low-value modal map and Sobel modal feature map corresponding to the nodule interior and boundary;

[0009] Step 3: According to the rating indicators specified in C-TIRADS, six diagnostic feature extraction methods were designed, using support vector machines or convolutional networks for classification;

[0010] Step 4: For the segmented nodule image, extract the required diagnostic features and classify them according to the situation. Finally, according to the rule that the presence of positive indicators is +1 and the presence of negative indicators is -1, obtain the final C-TIRADS rating result.

[0011] Furthermore, step 1 specifically includes:

[0012] Step 101: Use the double residual multi-convolution module I proposed in the present invention to replace the two consecutive 3*3 convolution layers with the same channel output in the U-Net network. o =M DR (I i ,4,2)replace, where M DR is the double residual multi-convolution module proposed in the present invention, 4 and 2 represent the number of forward convolutions and the number of residual connections respectively; first, the input I of each module i Four 3*3 convolutional layers are used, and the output of these four convolutional layers is T1=C(I i,3), T2=C(T1,3), T3=C(T2,3), T4=C(T3',3), where, T i represents the output of the i-th convolutional layer, C(T,n) represents the n*n convolution operation on the input T; then, these four outputs are connected in the channel dimension to obtain T=P(T1,T2,T3,T4), where P represents the channel dimension concatenation of the internal feature maps; in addition, in the designed module, two 1*1 convolutional layer residual modules are introduced before the fourth convolution and the final module output, so that T3'=T3+C(T1,1), I o =T+I i , where I o is the output of the double residual multi-convolution module;

[0013] Step 102: Before connecting the shallow features of the U-shaped structure with the deep features, the output I of each double residual multi-convolution module is o The GLCM modal feature map was extracted, and then the gray level co-occurrence matrix texture modal features of the thyroid nodule were extracted at four directions of 0°, 45°, 90° and 135°, namely, D w (I o )'=W(GL(I o )), where GL represents the extraction of GLCM modal feature map, and W represents the extraction of gray level co-occurrence matrix texture modal feature; finally, after mRMR feature screening, the features with a score not 0 are selected to obtain D w (I o )=M TF (I o )=mR(D w (I o )'), where M TF represents the texture modal feature extraction module, mR represents the mRMR feature screening operation; the 12 texture features obtained are as follows: contrast -0°, contrast -45°, contrast -90°, inverse difference moment -0°, entropy -0°, difference variance -0°, difference variance -45°, difference variance -90°, difference variance -135°, difference entropy -0°, difference entropy -90° and homogeneity -0°;

[0014] Step 103: The skip connection in the original U-Net network is replaced by the GLCM modal feature map supplementary connection with the efficient channel attention mechanism proposed in the present invention, that is, the gray level co-occurrence matrix texture modal feature D in step 102 is replaced by the gray level co-occurrence matrix texture modal feature D in step 102. w (I o ) After optimization of the efficient channel attention mechanism module, we get I eca (I o )=E(D w (I o), where E represents the attention mechanism module; then the output of the previous layer I i And the deep GLCM modal feature map I after 1*1 convolution and optimization eca (C(I o ,1)) Connect in the channel dimension to obtain the output of the GLCM modal feature map supplementary connection, that is, I q (I o )=M LM (I o )=P(I i ,I eca (C(I o ,1))), where M LM Indicates this supplementary connection module.

[0015] Furthermore, step 2 specifically includes:

[0016] Step 201: Use the nodule labels obtained by segmentation to extract the nodule part corresponding to the original image, and use it as the input of the automatic rating network after gray value mapping. r , extract the convex hull to obtain I = h (I r ), where h represents the convex hull extraction operation;

[0017] Step 202: perform corrosion operation on the nodule area to obtain the inner area I of the original image. in =ed(I,r), where ed represents the erosion operation and r represents the radius of the circular structure in the erosion operation; the Sobel modal feature map S(I in ), where S represents the extraction of Sobel modal feature map;

[0018] Step 203: Perform dilation operation on the nodule region to obtain I e =ep(I,r), where ep represents the dilation operation; the internal area I obtained by performing the corrosion operation on the nodule area in2 =ed(I,r); remove the internal area of ​​the expanded area and finally obtain the boundary area I of the thyroid nodule bd , and extract the corresponding Sobel modal feature map S(I bd );

[0019] Step 204: thyroid nodule internal area I in and the boundary region I bd , where any pixel is replaced by the minimum grayscale value in the neighborhood of the pixel, and the resulting image is the neighborhood low-value modal image m(I in ,r') and m(I bd ,r'), where m represents the grayscale value transformation operation mentioned above and r' represents the radius of the neighborhood.

[0020] Furthermore, step 3 specifically includes:

[0021] Step 301: extract the vertical position index, calculate the aspect ratio of the image I, and classify it using a support vector machine, that is, Among them, SVM means using support vector machine for classification, l L_ch and l T_ch Respectively represent the longest longitudinal axis and the longest transverse axis of the convex hull;

[0022] Step 302: extract solidity index and analyze the internal area of ​​thyroid nodule I in and the corresponding Sobel mode diagram S(I in ), combined with the gray mean F ave , standard deviation F std , Kurtosis F kut and entropy F ent Features, using a support vector machine for classification, the formula is as follows:

[0023]

[0024] Among them, N ch_eraode Indicates the number of pixels in the convex hull after corrosion, i n Represents the gray value of the nth pixel in the convex hull after corrosion, p i It represents the probability of the pixel with gray value i appearing in the convex hull after corrosion. The superscripts a and b represent the internal area of ​​the nodule and the corresponding Sobel mode map features, respectively.

[0025] Step 303: Extract the obvious hypoechoic index and analyze the inner area I of the original image of the thyroid nodule. in and the boundary region I bd , and the corresponding neighborhood low-value mode graph m(I in ,r'),m(I bd ,r'), extract the minimum value, mean, standard deviation, kurtosis and entropy features, and for the corresponding Sobel mode diagram S(I in )、S(I bd ), extract the mean, standard deviation, kurtosis and entropy features, and use support vector machine training respectively, the method is the same as in step 33, and obtain three obvious hypoechoic classification results c31, c32, c33;

[0026] Step 304: extract microcalcification indicators, perform corrosion operations on the convex hull and extract the MSER maximum stable extreme value region. If there is microcalcification, dilate the maximum stable extreme value region and exclude pixels with smaller gray values. Then extract its mean, standard deviation, kurtosis and entropy features. Based on the above features, train a classifier with a fully connected layer parameter of 100 to obtain the classification result, namely:

[0027]

[0028] Among them, J represents the use of convolution for classification, N mc represents the maximum number of stable extreme value regions, S mc represents the area of ​​each region, and 100 represents the number of parameters of the full convolutional layer;

[0029] Step 305: extract edge blur / irregularity / external invasion features, use the Sobel modal feature map of the image to extract features for the thyroid nodule boundary, interior, and overall, extract the standard deviation, and take the internal and external variance F cv Equitable divisibility F bou_in ; The above features were used to train a fully connected classifier with a layer parameter of 10 to obtain the classification results for thyroid nodule fuzziness, that is,

[0030]

[0031] Among them, N Sbd and G Sbd Respectively represent the total number of pixels and grayscale value of the boundary Sobel modal feature map, and N Sin and G Sin Respectively represent the total number of pixels and grayscale value of the internal Sobel modal feature map, N a Represents the total number of pixels of the overall Sobel modal feature map;

[0032] Extract the following features of nodules: compactness Rectangle fitting factor and eccentricity characteristics The ratio of the convex hull area to the segmentation result area F s Ratio of perimeter to F l , using support vector machine for classification, we get the irregularity and external invasion indicators of nodules, namely:

[0033]

[0034] Step 306: Extract the comet tail artifact features. Assuming that a certain connected area has a maximum of x pixels in the horizontal direction, the comet tail simulation area is below the connected area, with x+2 pixels as the base, and forms an isosceles triangle area in the vertical direction as the comet tail simulation area I. m; The standard deviation of the corresponding thyroid nodule ultrasound original image is calculated, and the comet tail artifact index is obtained using the support vector machine, namely:

[0035] c6=SVM(I m ,I).

[0036] Furthermore, step 4 specifically includes:

[0037] Step 401: extract the vertical index, solid index, microcalcification index and edge blur / irregularity / external invasion index of the segmented nodule image in turn, and classify them;

[0038] Step 402, extracting and classifying three indicators of obvious hypoechoic sound, and integrating the obtained c51, c52, and c53 with the assistance of the solidity indicator in step 41 to obtain the final classification result c5;

[0039] Step 403: when extracting and classifying microcalcification indicators in step 401, if there is still a maximum stable extreme value region after excluding pixels with smaller gray values, but the classification result is that there is no microcalcification, the comet tail indicator will be extracted and classified;

[0040] Step 404: According to the rule that the existence of a positive indicator is +1 and the existence of a negative indicator is -1, the final C-TIRADS rating results are obtained and divided into 2, 3, 4A, 4B, 4C and 5 categories.

[0041] The beneficial effects of the present invention are:

[0042] 1. Aiming at the key points of segmentation of ultrasound images of thyroid nodules, the present invention proposes a method for improving the U-Net network by using the proposed deep GLCM modal feature map supplementary connection based on the attention mechanism and the multi-convolution module based on double residual improvement. Compared with the single U-Net, the method of the present invention can not only extract the features of the original image, but also extract the GLCM modal feature map and the supplementary connection module, and introduce the double residual multi-convolution module to enrich the structure of the network and the extraction of features, which can significantly improve the accuracy of nodule segmentation when the number of samples is small, and achieve thyroid nodule segmentation with better performance in accuracy, shape and area estimation, thereby reducing the workload of doctors;

[0043] 2. The automatic rating part of the present invention extracts 6 diagnostic features based on C-TIRADS, quantifies specific rating indicators, and realizes fully automated thyroid nodule rating; the extracted features are authoritative, and each indicator has a high accuracy rate and reasonable interpretability, which can be understood and recognized by doctors; the rating process takes into account the relationship between various indicators and makes full use of them, and the results obtained are more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the method for intelligent segmentation and automatic rating of thyroid ultrasound images of the present invention;

[0045] Figure 2 This is a schematic diagram of the network structure of GADRU-Net;

[0046] Figure 3 It is a schematic diagram of the supplementary connection structure based on the GLCM modal feature map;

[0047] Figure 4 It is a schematic diagram of the structure of the double residual multi-convolution module;

[0048] Figure 5 This is an overlay of thyroid nodule segmentation results based on GADRU-Net and U-Net;

[0049] Figure 6 It is a flow chart for quantification of microcalcification index in C-TIRADS of thyroid nodules;

[0050] Figure 7 It is a quantitative diagram of all rating indicators based on C-TIRADS;

[0051] Figure 8 This is a diagram of the confusion matrix based on the C-TIRADS quantitative rating results. DETAILED DESCRIPTION

[0052] Specific implementation method 1: Figures 1 to 8 As shown in FIG. 1 , a method for intelligent segmentation and automatic rating of thyroid nodules based on ultrasound images is provided. Step 1: using the proposed thyroid nodule segmentation method based on the GADRU-Net network, according to the thyroid ultrasound images I of the training set T and the corresponding segmentation label L IT Train the network and get GR(M DR ,M TF ,M LM )(I T ,L IT ), where GR is the proposed GADRU-Net segmentation network, M DR 、M LM and M TF They are the dual residual multi-convolution module, texture modality feature extraction module and supplementary connection module in the segmentation network; then the test set image I V Perform segmentation and obtain segmented nodule labels Step 1 specifically includes:

[0053] Step 101: Use the double residual multi-convolution module I proposed in the present invention to replace the two consecutive 3*3 convolution layers with the same channel output in the U-Net network. o =MDR (I i ,4,2)replace, where M DR is the double residual multi-convolution module proposed in the present invention, 4 and 2 represent the number of forward convolutions and the number of residual connections respectively; first, the input I of each module i Four 3*3 convolutional layers are used, and the output of these four convolutional layers is T1=C(I i ,3), T2=C(T1,3), T3=C(T2,3), T4=C(T3',3), where, T i represents the output of the i-th convolutional layer, C(T,n) represents the n*n convolution operation on the input T; then, these four outputs are connected in the channel dimension to obtain T=P(T1,T2,T3,T4), where P represents the channel dimension concatenation of the internal feature maps; in addition, in the designed module, two 1*1 convolutional layer residual modules are introduced before the fourth convolution and the final module output, so that T3'=T3+C(T1,1), I o =T+I i , where I o is the output of the double residual multi-convolution module;

[0054] Step 102: Before connecting the shallow features of the U-shaped structure with the deep features, the output I of each double residual multi-convolution module is o The GLCM modal feature map was extracted, and then the gray level co-occurrence matrix texture modal features of the thyroid nodule were extracted at four directions of 0°, 45°, 90° and 135°, namely, D w (I o )'=W(GL(I o )), where GL represents the extraction of GLCM modal feature map, and W represents the extraction of gray level co-occurrence matrix texture modal feature; finally, after mRMR feature screening, the features with a score not 0 are selected to obtain D w (I o )=M TF (I o )=mR(D w (I o )'), where M TF represents the texture modal feature extraction module, mR represents the mRMR feature screening operation; the 12 texture features obtained are as follows: contrast -0°, contrast -45°, contrast -90°, inverse difference moment -0°, entropy -0°, difference variance -0°, difference variance -45°, difference variance -90°, difference variance -135°, difference entropy -0°, difference entropy -90° and homogeneity -0°;

[0055] Step 103: The skip connection in the original U-Net network is replaced by the GLCM modal feature map supplementary connection with the efficient channel attention mechanism proposed in the present invention, that is, the gray level co-occurrence matrix texture modal feature D in step 102 is replaced by the gray level co-occurrence matrix texture modal feature D in step 102. w (I o ) After optimization of the efficient channel attention mechanism module, we get I eca (I o )=E(D w (I o ), where E represents the attention mechanism module; then the output of the previous layer I i And the deep GLCM modal feature map I after 1*1 convolution and optimization eca (C(I o ,1)) Connect in the channel dimension to obtain the output of the GLCM modal feature map supplementary connection, that is, I q (I o )=M LM (I o )=P(I i ,I eca (C(I o ,1))), where M LM represents this supplementary connection module;

[0056] Step 2: Use the nodule labels obtained by segmentation to extract the nodule part corresponding to the original ultrasound image, and perform preprocessing operations, including gray value mapping, convex hull extraction, nodule internal area extraction, nodule boundary area extraction, and extraction of the neighborhood low-value modal map and Sobel modal feature map corresponding to the nodule interior and boundary; Step 2 specifically includes:

[0057] Step 201: Use the nodule labels obtained by segmentation to extract the nodule part corresponding to the original image, and use it as the input of the automatic rating network after gray value mapping. r , extract the convex hull to obtain I = h (I r ), where h represents the convex hull extraction operation;

[0058] Step 202: perform corrosion operation on the nodule area to obtain the inner area I of the original image. in =ed(I,r), where ed represents the erosion operation and r represents the radius of the circular structure in the erosion operation; the Sobel modal feature map S(I in ), where S represents the extraction of Sobel modal feature map;

[0059] Step 203: Perform dilation operation on the nodule region to obtain I e =ep(I,r), where ep represents the dilation operation; the internal area I obtained by performing the corrosion operation on the nodule areain2 =ed(I,r); remove the internal area of ​​the expanded area and finally obtain the boundary area I of the thyroid nodule bd , and extract the corresponding Sobel modal feature map S(I bd );

[0060] Step 204: thyroid nodule internal area I in and the boundary region I bd , where any pixel is replaced by the minimum grayscale value in the neighborhood of the pixel, and the resulting image is the neighborhood low-value modal image m(I in ,r') and m(I bd ,r'), where m represents the grayscale value transformation operation mentioned above, and r' represents the radius of the neighborhood;

[0061] Step 3: According to the rating indicators specified in C-TIRADS, six diagnostic feature extraction methods were designed, and support vector machines or convolutional networks were used for classification. Step 3 specifically includes:

[0062] Step 301: extract the vertical position index, calculate the aspect ratio of the image I, and classify it using a support vector machine, that is, Among them, SVM means using support vector machine for classification, l L_ch and l T_ch Respectively represent the longest longitudinal axis and the longest transverse axis of the convex hull;

[0063] Step 302: extract solidity index and analyze the internal area of ​​thyroid nodule I in and the corresponding Sobel mode diagram S(I in ), combined with the gray mean F ave , standard deviation F std , Kurtosis F kut and entropy F ent Features, using a support vector machine for classification, the formula is as follows:

[0064]

[0065] Among them, N ch_eraode Indicates the number of pixels in the convex hull after corrosion, i n Represents the gray value of the nth pixel in the convex hull after corrosion, p i It represents the probability of the pixel with gray value i appearing in the convex hull after corrosion. The superscripts a and b represent the internal area of ​​the nodule and the corresponding Sobel mode map features, respectively.

[0066] Step 303: Extract the obvious hypoechoic index and analyze the inner area I of the original image of the thyroid nodule. in and the boundary region I bd, and the corresponding neighborhood low-value mode graph m(I in ,r'),m(I bd ,r'), extract the minimum value, mean, standard deviation, kurtosis and entropy features, and for the corresponding Sobel mode diagram S(I in )、S(I bd ), extract the mean, standard deviation, kurtosis and entropy features, and use support vector machine training respectively, the method is the same as in step 33, and obtain three obvious hypoechoic classification results c31, c32, c33;

[0067] Step 304: extract microcalcification indicators, perform corrosion operations on the convex hull and extract the MSER maximum stable extreme value region. If there is microcalcification, dilate the maximum stable extreme value region and exclude pixels with smaller gray values. Then extract its mean, standard deviation, kurtosis and entropy features. Based on the above features, train a classifier with a fully connected layer parameter of 100 to obtain the classification result, namely:

[0068]

[0069] Among them, J represents the use of convolution for classification, N mc represents the maximum number of stable extreme value regions, S mc represents the area of ​​each region, and 100 represents the number of parameters of the full convolutional layer;

[0070] Step 305: extract edge blur / irregularity / external invasion features, use the Sobel modal feature map of the image to extract features for the thyroid nodule boundary, interior, and overall, extract the standard deviation, and take the internal and external variance F cv Equitable divisibility F bou_in ; The above features were used to train a fully connected classifier with a layer parameter of 10 to obtain the classification results for thyroid nodule fuzziness, that is,

[0071]

[0072] Among them, N Sbd and G Sbd Respectively represent the total number of pixels and grayscale value of the boundary Sobel modal feature map, and N Sin and G Sin Respectively represent the total number of pixels and grayscale value of the internal Sobel modal feature map, N a Represents the total number of pixels of the overall Sobel modal feature map;

[0073] Extract the following features of nodules: compactness Rectangle fitting factor and eccentricity characteristics The ratio of the convex hull area to the segmentation result area F sRatio of perimeter to F l , using support vector machine for classification, we get the irregularity and external invasion indicators of nodules, namely:

[0074]

[0075] Step 306: Extract the comet tail artifact features. Assuming that a certain connected area has a maximum of x pixels in the horizontal direction, the comet tail simulation area is below the connected area, with x+2 pixels as the base, and forms an isosceles triangle area in the vertical direction as the comet tail simulation area I. m ; The standard deviation of the corresponding thyroid nodule ultrasound original image is calculated, and the comet tail artifact index is obtained using the support vector machine, namely:

[0076] c6=SVM(I m ,I);

[0077] Step 4: For the segmented nodule image, extract the required diagnostic features and classify them according to the situation, and finally obtain the final C-TIRADS rating result according to the rule that the existence of positive indicators is +1 and the existence of negative indicators is -1; Step 4 specifically includes:

[0078] Step 401: extract the vertical index, solid index, microcalcification index and edge blur / irregularity / external invasion index of the segmented nodule image in turn, and classify them;

[0079] Step 402, extracting and classifying three indicators of obvious hypoechoic sound, and integrating the obtained c51, c52, and c53 with the assistance of the solidity indicator in step 41 to obtain the final classification result c5;

[0080] Step 403: when extracting and classifying microcalcification indicators in step 401, if there is still a maximum stable extreme value region after excluding pixels with smaller gray values, but the classification result is that there is no microcalcification, the comet tail indicator will be extracted and classified;

[0081] Step 404: According to the rule that the existence of a positive indicator is +1 and the existence of a negative indicator is -1, the final C-TIRADS rating results are obtained and divided into 2, 3, 4A, 4B, 4C and 5 categories.

[0082] Among them, positive indicators refer to vertical position indicators, solid indicators, obvious low echo indicators, microcalcification indicators, and blurred / irregular / external invasion indicators; negative indicators refer to comet tail indicators.

[0083] Example

[0084] The original thyroid nodule ultrasound images and corresponding segmentation labels used in the experiment were provided by the Heilongjiang Key Laboratory of Trace Elements and the Endemic Disease Control Center of Harbin Medical University, and verified by professional physicians in the ultrasound department. A total of 1,154 thyroid nodule samples were collected, and the benign and malignant thyroid nodule samples were divided at a ratio of approximately 0.75:0.25. The resulting training set consisted of 426 benign samples and 449 malignant samples, while the test set consisted of 126 benign samples and 153 malignant samples.

[0085] Execute step 1: Design GADRU-Net as Figure 2 As shown in, the supplementary connection module and the double residual multi-convolution module are as follows Figure 3 , 4 Before the network is trained, the network's hyperparameters are set: momentum is set to 0.9, weight decay is set to 1e-08, and learning rate is set to 1e-05 to 1e-06.

[0086] GADRU-Net was trained using the RMSprop optimizer, and then the divided test set was segmented. The results were compared with their segmentation labels and the segmentation results of the traditional U-Net network. The overlay of the thyroid nodule segmentation results based on the GADRU-Net network and the traditional U-Net network is shown in Figure 2. Figure 5 The mean values ​​of the segmentation accuracy evaluation indicators of the segmentation results are shown in Table 1.

[0087] Table 1 Mean values ​​of segmentation accuracy evaluation indicators based on GADRU-Net and U-Net segmentation results

[0088]

[0089] from Figure 5 It can be seen intuitively that the thyroid nodule segmentation results obtained by the GADRU-Net are mostly very similar to the segmentation boundary lines. Among them, some results are significantly improved by the GADRU-Net network in large areas that were not originally segmented by the U-Net network. There are also some more accurate results in the segmentation of the corner areas of the thyroid nodules. As can be seen from Table 1, the GADRU-Net has an advantage of 1.21% in IoU, 1.22% in MCC, and 1.09% in F1 compared to U-Net. Therefore, it can be concluded that the GADRU-Net network is superior to the traditional U-Net network in thyroid nodule segmentation.

[0090] Execute step 2: Use the nodule label obtained by segmentation to extract the nodule part corresponding to the original ultrasound image; extract the convex hull after gray value mapping; use a circular structure element with a radius of 5 to perform an erosion operation to obtain the internal area of ​​the original image; extract the Sobel modal feature map of the corresponding part by integrating the horizontal and vertical edge maps; use a circular structure element with a radius of 9 to perform an expansion operation, subtract the internal area obtained by the erosion operation using a circular structure element with a radius of 3, remove the internal area from the expansion area, obtain the boundary area of ​​the thyroid nodule, and extract the corresponding Sobel modal feature map; for the internal area and boundary area of ​​the thyroid nodule, any pixel in the original image is replaced by the minimum gray value in the 5*5 neighborhood centered on the pixel, and the resulting image is the neighborhood low-value modal map;

[0091] Execute step three: A total of 852 thyroid nodule ultrasound samples in the data include specific classification of vertical position, solidity, obvious hypoechoic, microcalcification, shape regularity and comet tail artifacts. The samples that can be used for automatic rating research are divided at a ratio of approximately 0.8:0.2, and a rating training set based on segmentation labels and a rating test set based on segmentation results are respectively formed. Each data set is then graded according to the C-TIRADS indicators; the specific rating is completed and verified by the ultrasound experts of the tertiary hospital who provided the data source.

[0092] Using the divided training set, six diagnostic features were extracted for nodules: vertical position index was extracted, and the aspect ratio was extracted on the extracted convex hull image, and a support vector machine was used for classification; solidity index was extracted, and the original ultrasound image and Sobel modal image of the thyroid nodule corresponding to the eroded convex hull were combined with the grayscale mean, standard deviation, kurtosis and entropy features, and a support vector machine was used for classification;

[0093] Extract obvious hypoechoic indexes, extract the minimum value, mean, standard deviation, kurtosis and entropy of the internal area and boundary area of ​​the original image of thyroid nodules and the neighborhood low-value modal map, extract the mean, standard deviation, kurtosis and entropy of the corresponding Sobel modal map, and then use support vector machines to train 10 features from the original image, 8 features from the Sobel modal feature map, and 10 features from the neighborhood low-value modal map to obtain three obvious hypoechoic classification results;

[0094] Extract microcalcification indicators and quantify the process as follows Figure 6 As shown in the figure, firstly, the convex hull is corroded, then the maximum stable extreme value region of the MSER is extracted for the thyroid nodule ultrasound image, and the connected area with small changes under different threshold segmentation is retained, and then only the pixel size of 2 is retained. 2 -7 2If the maximum stable extreme value region cannot be extracted from the corresponding image, it means that there is no microcalcification. If there is microcalcification, the maximum stable extreme value region is expanded using a 2*2 square structure element, and the pixels with grayscale values ​​less than 150 in the corresponding original image are excluded from the maximum stable extreme value region. If there is no maximum stable extreme value region at all, it is determined that the corresponding image does not have microcalcification. In order to distinguish whether there is microcalcification, the mean, standard deviation, kurtosis and entropy features are extracted, and a classifier with a fully connected layer parameter of 100 is trained to obtain the classification results.

[0095] The edge blur / irregularity / external invasion indicators were extracted, and the Sobel modal feature map of the image was used to extract features for the boundary, interior and overall of the thyroid nodule. The standard deviation was extracted based on the Sobel modal feature maps of the boundary, interior and overall regions, and the internal and external variance and even separability were taken. A fully connected classifier with a layer parameter of 10 was trained based on the above five features to obtain the classification results for the blurred boundary of the thyroid nodule; then, the compactness, rectangle fitting factor and eccentricity features were extracted, as well as the area ratio and perimeter ratio of the region after convex hull extraction to the segmentation result / segmentation label, and the support vector machine was used to obtain the classification results of irregular and external invasion features.

[0096] To extract the comet tail artifact features, firstly, the maximum stable extreme value region of the thyroid nodule ultrasound image is extracted in the same way as in the calcification index quantification process. Then, the maximum stable extreme value region is also expanded using a 2*2 square structure element, and only pixels with grayscale values ​​less than 128 are excluded from the maximum stable extreme value region. For each connected area in the remaining reserved area, it is assumed that a certain connected area is a maximum of x pixels in the horizontal direction, with x+2 pixels as the base, and 2 pixels are reduced in the horizontal direction for every 1 pixel deep on the vertical axis to form an isosceles triangle area. The grayscale value of the base in the triangle area is defined as the highest grayscale value of the thyroid nodule ultrasound original image area corresponding to the area, and the grayscale value of the area with a depth of 1 pixel is defined as 0.9 times that of the previous layer, and the comet tail simulation area is obtained. The standard deviation of the comet tail simulation area and its corresponding thyroid nodule ultrasound original image is calculated, and finally the support vector machine is used to obtain the C-TIRADS comet tail artifact index.

[0097] Execute step four, extract the vertical position index, solidity index, microcalcification index and edge blur / irregularity / external invasion index of the segmented nodule image in turn, and classify them; extract the three indexes of obvious low echo and classify them, and integrate the obtained features with the assistance of the solidity index in step four-one to obtain the final classification result; when extracting and classifying microcalcification indexes, if there is still a maximum stable extreme value area after excluding pixels with smaller grayscale values, but the classification result is that there is no microcalcification, the comet tail index will be extracted and classified; according to the rule that the existence of positive indicators (vertical position indicators, solidity indicators, obvious low echo indicators, microcalcification indicators, edge blur / irregularity / external invasion indicators) is +1, and the existence of negative indicators (comet tail indicators) is -1, the final C-TIRADS rating results are obtained, which are divided into 2, 3, 4A, 4B, 4C and 5 categories. The automatic rating process is as follows: Figure 7 shown.

[0098] The test set is used for automatic rating test. The accuracy of the six evaluation indicators is shown in Table 2. The confusion matrix of the quantitative rating results is shown in Figure 8 As shown. After analyzing all the above six indicators one by one, it can be found that most of the indicators can achieve an accuracy rate of more than 90% after quantification. Finally, after calculating the overall C-TIRADS rating, most thyroid nodules can be correctly rated with an accuracy rate of 79.88%.

[0099] Table 2 Accuracy of evaluation indicators

[0100]

[0101] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for intelligent segmentation and automatic rating of thyroid nodules based on ultrasound images, characterized in that: The specific steps include: Step 1: Using the proposed GADRU-Net network-based thyroid nodule segmentation method, according to the thyroid ultrasound images of the training set I T and the corresponding segmentation labels Train the network and get Among them, GR is the proposed GADRU-Net segmentation network, M DR 、M LM and M TF They are the dual residual multi-convolution module, texture modality feature extraction module and supplementary connection module in the segmentation network; then the test set image I V Perform segmentation and obtain segmented nodule labels Step 2: Use the nodule labels obtained by segmentation to extract the nodule part corresponding to the original ultrasound image, and perform preprocessing operations, including gray value mapping, convex hull extraction, nodule internal area extraction, nodule boundary area extraction, and extraction of the neighborhood low-value modal map and Sobel modal feature map corresponding to the nodule interior and boundary; Step 3: According to the rating indicators specified in C-TIRADS, six diagnostic feature extraction methods were designed, using support vector machines or convolutional networks for classification; Step 4: For the segmented nodule image, extract the required diagnostic features and classify them according to the situation. Finally, according to the rule that the presence of positive indicators is +1 and the presence of negative indicators is -1, obtain the final C-TIRADS rating result.

2. The method for intelligent segmentation and automatic rating of thyroid nodules based on ultrasound images according to claim 1, characterized in that: Step 1 specifically includes: Step 101: Use the double residual multi-convolution module I proposed in the present invention to replace the two consecutive 3*3 convolution layers with the same channel output in the U-Net network. o =M DR (I i ,4,2)replace, where M DR is the double residual multi-convolution module proposed in the present invention, 4 and 2 represent the number of forward convolutions and the number of residual connections respectively; first, the input I of each module i Four 3*3 convolutional layers are used, and the output of these four convolutional layers is T1=C(I i ,3), T2=C(T1,3), T3=C(T2,3), T4=C(T3',3), where, T i represents the output of the i-th convolutional layer, C(T,n) represents the n*n convolution operation on the input T; then, these four outputs are connected in the channel dimension to obtain T=P(T1,T2,T3,T4), where P represents the channel dimension concatenation of the internal feature maps; in addition, in the designed module, two 1*1 convolutional layer residual modules are introduced before the fourth convolution and the final module output, so that T3'=T3+C(T1,1), I o =T+I i , where I o is the output of the double residual multi-convolution module; Step 102: Before connecting the shallow features of the U-shaped structure with the deep features, the output I of each double residual multi-convolution module is o The GLCM modal feature map was extracted, and then the gray level co-occurrence matrix texture modal features of the thyroid nodule were extracted at four directions of 0°, 45°, 90° and 135°, namely, D w (I o )'=W(GL(I o )), where GL represents the extraction of GLCM modal feature map, and W represents the extraction of gray level co-occurrence matrix texture modal feature; finally, after mRMR feature screening, the features with a score not 0 are selected to obtain D w (I o )=M TF (I o )=mR(D w (I o )'), where M TF represents the texture modal feature extraction module, mR represents the mRMR feature screening operation; the 12 texture features obtained are as follows: contrast -0°, contrast -45°, contrast -90°, inverse difference moment -0°, entropy -0°, difference variance -0°, difference variance -45°, difference variance -90°, difference variance -135°, difference entropy -0°, difference entropy -90° and homogeneity -0°; Step 103: The skip connection in the original U-Net network is replaced by the GLCM modal feature map supplementary connection with the efficient channel attention mechanism proposed in the present invention, that is, the gray level co-occurrence matrix texture modal feature D in step 102 is replaced by the gray level co-occurrence matrix texture modal feature D in step 102. w (I o ) After optimization of the efficient channel attention mechanism module, we get I eca (I o )=E(D w (I o ), where E represents the attention mechanism module; then the output of the previous layer I i And the deep GLCM modal feature map I after 1*1 convolution and optimization eca (C(I o ,1)) Connect in the channel dimension to obtain the output of the GLCM modal feature map supplementary connection, that is, I q (I o )=M LM (I o )=P(I i ,I eca (C(I o ,1))), where M LM Indicates this supplementary connection module.

3. The method for intelligent segmentation and automatic rating of thyroid nodules based on ultrasound images according to claim 1, characterized in that: Step 2 specifically includes: Step 201: Use the nodule labels obtained by segmentation to extract the nodule part corresponding to the original image, and use it as the input of the automatic rating network after gray value mapping. r , extract the convex hull to obtain I = h (I r ), where h represents the convex hull extraction operation; Step 202: perform corrosion operation on the nodule area to obtain the inner area I of the original image. in =ed(I,r), where ed represents the erosion operation and r represents the radius of the circular structure in the erosion operation; the Sobel modal feature map S(I in ), where S represents the extraction of Sobel modal feature map; Step 203: Perform dilation operation on the nodule region to obtain I e =ep(I,r), where ep represents the dilation operation; the internal area I obtained by performing the corrosion operation on the nodule area in2 =ed(I,r); remove the internal area of ​​the expanded area and finally obtain the boundary area I of the thyroid nodule bd , and extract the corresponding Sobel modal feature map S(I bd ); Step 204: thyroid nodule internal area I in and the boundary region I bd , where any pixel is replaced by the minimum grayscale value in the neighborhood of the pixel, and the resulting image is the neighborhood low-value modal image m(I in ,r') and m(I bd ,r'), where m represents the grayscale value transformation operation mentioned above and r' represents the radius of the neighborhood.

4. The method for intelligent segmentation and automatic rating of thyroid nodules based on ultrasound images according to claim 1, characterized in that: Step 3 specifically includes: Step 301: extract the vertical position index, calculate the aspect ratio of the image I, and classify it using a support vector machine, that is, Among them, SVM means using support vector machine for classification, l L_ch and l T_ch Respectively represent the longest longitudinal axis and the longest transverse axis of the convex hull; Step 302: extract solidity index and analyze the internal area of ​​thyroid nodule I in and the corresponding Sobel mode diagram S(I in ), combined with the gray mean F ave , standard deviation F std , Kurtosis F kut and entropy F ent Features, using a support vector machine for classification, the formula is as follows: Among them, N ch_eraode Indicates the number of pixels in the convex hull after corrosion, i n Represents the gray value of the nth pixel in the convex hull after corrosion, p i It represents the probability of the pixel with gray value i appearing in the convex hull after corrosion. The superscripts a and b represent the internal area of ​​the nodule and the corresponding Sobel mode map features, respectively. Step 303: Extract the obvious hypoechoic index and analyze the inner area I of the original image of the thyroid nodule. in and the boundary region I bd , and the corresponding neighborhood low-value mode graph m(I in ,r')、m(I bd ,r'), extract the minimum value, mean, standard deviation, kurtosis and entropy features, and for the corresponding Sobel mode diagram S(I in )、S(I bd ), extract the mean, standard deviation, kurtosis and entropy features, and use support vector machine training respectively, the method is the same as in step 33, and obtain three obvious hypoechoic classification results c31, c32, c33; Step 304: extract microcalcification indicators, perform corrosion operations on the convex hull and extract the MSER maximum stable extreme value region. If there is microcalcification, dilate the maximum stable extreme value region and exclude pixels with smaller gray values. Then extract its mean, standard deviation, kurtosis and entropy features. Based on the above features, train a classifier with a fully connected layer parameter of 100 to obtain the classification result, namely: Among them, J represents the use of convolution for classification, N mc represents the maximum number of stable extreme value regions, S mc represents the area of ​​each region, and 100 represents the number of parameters of the full convolutional layer; Step 305: extract edge blur / irregularity / external invasion features, use the Sobel modal feature map of the image to extract features for the thyroid nodule boundary, interior, and overall, extract the standard deviation, and take the internal and external variance F cv Equitable divisibility F bou_in ; The above features were used to train a fully connected classifier with a layer parameter of 10 to obtain the classification results for thyroid nodule fuzziness, that is, Among them, N Sbd and G Sbd Respectively represent the total number of pixels and grayscale value of the boundary Sobel modal feature map, and N Sin and G Sin Respectively represent the total number of pixels and grayscale value of the internal Sobel modal feature map, N a Represents the total number of pixels of the overall Sobel modal feature map; Extract the following features of nodules: compactness Rectangle fitting factor and eccentricity characteristics The ratio of the convex hull area to the segmentation result area F s Ratio of perimeter to F l , using support vector machine for classification, we get the irregularity and external invasion indicators of nodules, namely: Step 306: Extract the comet tail artifact features. Assuming that a certain connected area has a maximum of x pixels in the horizontal direction, the comet tail simulation area is below the connected area, with x+2 pixels as the base, and forms an isosceles triangle area in the vertical direction as the comet tail simulation area I. m ; The standard deviation of the corresponding thyroid nodule ultrasound original image is calculated, and the comet tail artifact index is obtained using the support vector machine, namely: c6=SVM(I m ,I)。 5. The method for intelligent segmentation and automatic rating of thyroid nodules based on ultrasound images according to claim 1, characterized in that: Step 4 specifically includes: Step 401: extract the vertical index, solid index, microcalcification index and edge blur / irregularity / external invasion index of the segmented nodule image in turn, and classify them; Step 402, extracting and classifying three indicators of obvious hypoechoic sound, and integrating the obtained c51, c52, and c53 with the assistance of the solidity indicator in step 41 to obtain the final classification result c5; Step 403: when extracting and classifying microcalcification indicators in step 401, if there is still a maximum stable extreme value region after excluding pixels with smaller gray values, but the classification result is that there is no microcalcification, the comet tail indicator will be extracted and classified; Step 404: According to the rule that the existence of a positive indicator is +1 and the existence of a negative indicator is -1, the final C-TIRADS rating results are obtained and divided into 2, 3, 4A, 4B, 4C and 5 categories.

Images