A multi-scale and multi-region prediction method for thyroid nodules

Abstract

The present invention discloses a multi-scale and multi-region prediction method for thyroid nodules, belonging to the technical fields of medical image processing and deep learning; it includes: preparing a training data set; constructing a feature extraction model; constructing a feature reconstruction model; determining a prediction model; the present invention adopts ultrasonic images of different scales and different regions as inputs, combines high-frequency and low-frequency modules and a multi-scale feature extraction module. Specifically, during the network training process, the feature map is divided into high-frequency and low-frequency features in the channel direction to extract features of thyroid nodules, and then feature reconstruction is performed to obtain discriminative features; the thyroid nodule features obtained by this network are more comprehensive, and the accuracy of the benign and malignant prediction results is higher.

Description

Technical Field

[0001] The present invention belongs to the technical fields of medical image processing and deep learning, and particularly relates to a multi-scale and multi-region thyroid nodule prediction method. Background Art

[0002] The non-invasive diagnosis of thyroid nodules mainly relies on imaging examinations. Among them, ultrasound examination has the advantages of strong economy, harmlessness, non-invasiveness, simple operation and relatively high accuracy. However, the traditional ultrasound examination is affected by factors such as the professionalism of operators and the level of the medical environment, and the accuracy of ultrasound diagnosis is limited. With the continuous integration and development of computer and imaging technologies, deep learning methods have been widely applied to the imaging-assisted diagnosis of clinical diseases due to their advantages such as low consumption, high efficiency, high homogeneity and high accuracy. With the assistance of deep learning technology, different doctors can quantitatively analyze ultrasound images and form a unified diagnostic result report, reducing the influence of external factors such as doctor level and imaging equipment differences on the results. However, the main problem currently causing the low accuracy of classification results is that the nodule features in thyroid nodule ultrasound images are not extracted comprehensively enough.

[0003] The Chinese patent publication number is "CN201911271119", and the name is "A Thyroid Nodule Classification Method Based on Multi-Scale Feature Fusion". This method mainly cleans data, focuses on processing the data set, adds a high-resolution channel based on the residual network, and replaces the residual module with a multi-scale information fusion module. This method only inputs a single-pixel image and only has a high-resolution channel, and the feature extraction is not comprehensive enough. Summary of the Invention

[0004] In order to solve the problem of the low accuracy of the prediction results of the benign and malignant of thyroid nodules and the neglect of the overall features of nodules in the feature extraction process, the present invention proposes a multi-scale and multi-region thyroid nodule prediction method; the present invention inputs ultrasound images of different scales and different regions, combines high and low frequency modules and multi-scale feature extraction modules. Specifically, during the network training process, the feature map is divided into high and low frequency features in the channel direction, the features of thyroid nodules are extracted, and then feature reconstruction is performed to obtain discriminative features; the thyroid nodule features obtained by this network are more comprehensively extracted, and the accuracy of the benign and malignant prediction results is higher.

[0005] The present invention is realized through the following technical solutions:

[0006] A multi-scale and multi-region thyroid nodule prediction method, and the specific steps are as follows:

[0007] Step 1: Prepare a training data set: collect the thyroid ultrasound image set of thyroid patients, perform image preprocessing, and construct a training data set;

[0008] Step 2: Construct a feature extraction model: The feature extraction model includes a high-frequency and low-frequency feature extraction module and a multi-scale feature extraction module; input thyroid nodule images of different scales and different regions into the high-frequency and low-frequency feature extraction module, extract features from three aspects: nodule contour, nodule shape, and nodule edge, and use the output result as the input of the multi-scale feature extraction module for further feature extraction;

[0009] Step 3: Construct a feature reconstruction model: The feature reconstruction model performs feature reconstruction on the multi-scale feature extraction results obtained in Step 2, optimizes the network model, and improves the classification accuracy;

[0010] Step 4: Determine the prediction model: Use the training set to train the network model. After the number of training times reaches the preset threshold, use the test set to test the trained model. When the accuracy of the network model stabilizes within a certain established range, it is considered that the network model training is completed, and the parameters of the network model are saved; after the network training is completed, fix the network model parameters and determine the network model as the final nodule prediction model; finally, based on the trained network model, predict the type of thyroid nodule.

[0011] Further, in Step 1, the image set refers to thyroid nodule ultrasound images that have been confirmed as benign or malignant through puncture pathology during thyroid ultrasound examinations in the hospital; the image preprocessing includes removing invalid information, unifying the size, image normalization, and ultrasonic contrast enhancement, and dividing the training data set into a training set and a test set according to a ratio of 7:3.

[0012] Further, in Step 2, the high-frequency and low-frequency feature extraction module includes a high-frequency module and a low-frequency module. Among them, the high-frequency module is used to perform convolution and pooling operations on the data, and the low-frequency module is used to perform average pooling and upsampling operations on the data; the multi-scale feature extraction module includes four different convolutional blocks and a short-circuit mechanism.

[0013] Further, the four different convolutional blocks are respectively a convolutional layer, two convolutional layers, three convolutional layers, and four convolutional layers.

[0014] Further, in Step 3, the feature reconstruction model includes three skip modules and a convolutional layer; among them, each skip module includes four convolutional layers, four normalization operation layers, and four activation function layers.

[0015] Compared with the prior art, the advantages of the present invention are as follows:

[0016] 1. The present invention adopts a method of simultaneously extracting features from the low-frequency channel and the high-frequency channel, more comprehensively extracts the deep features and shallow features of the ultrasound image, effectively improves the prediction accuracy, and can better assist doctors in diagnosis in clinical medicine.

[0017] 2. The feature extraction part of the present invention inputs multi-scale thyroid nodule images, extracts features from different scales and different regions, can more comprehensively describe the nodule situation, and improve the prediction accuracy.

[0018] 3. The feature reconstruction part of the present invention adopts a skip connection structure and adds a convolutional layer, which can effectively increase the information flow, learn discriminative features of the image, avoid the problem of loss of important information caused by stacking convolutional layers, and improve the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0020] Figure 1 is a flowchart of a multi-scale and multi-region thyroid nodule prediction method of the present invention;

[0021] Figure 2 is a network structure diagram of a multi-scale and multi-region thyroid nodule prediction method of the present invention;

[0022] Figure 3 is a structure diagram of the high-low frequency feature extraction module of the present invention;

[0023] Figure 4 is a structure diagram of the multi-scale feature extraction module of the present invention;

[0024] Figure 5 is a structure diagram of the feature reconstruction part of the present invention;

[0025] Figure 6 is a structure diagram of the skip module of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] In order to clearly and completely describe the technical solutions of the present invention and its specific working process, in combination with the accompanying drawings of the specification, the specific embodiments of the present invention are as follows:

[0027] Example 1

[0028] As Figure 1 shown, it is a schematic flowchart of a multi-scale and multi-region thyroid nodule prediction method of this embodiment. The method specifically includes the following steps:

[0029] Step 1, Prepare the training dataset: Preprocess the thyroid ultrasound image set provided by the hospital to construct the training dataset. Among them, the image set refers to the ultrasound images of thyroid nodules that have been confirmed as benign or malignant through puncture pathology during thyroid ultrasound examinations in the hospital. The image preprocessing includes removing invalid information, unifying the size, image normalization, and enhancing the ultrasound contrast. The training data is divided into a training set and a test set according to a ratio of 7:3.

[0030] Step 2, Construct the feature extraction model: The feature extraction model includes a high-frequency and low-frequency feature extraction module and a multi-scale feature extraction module. Input thyroid nodule images of different scales and regions into the high-frequency and low-frequency feature extraction module, and extract features from three aspects: nodule contour, nodule shape, and nodule edge. Use the output result as the input of the multi-scale feature extraction module for further feature extraction. Among them, the high-frequency and low-frequency feature extraction module includes a high-frequency module and a low-frequency module. The high-frequency module includes convolution operations and pooling operations, and the low-frequency module includes average pooling operations, upsampling operations, and convolution operations. The multi-scale feature extraction module includes four convolutional blocks and a shortcut mechanism. Among them, the four convolutional blocks respectively contain one convolutional layer, two convolutional layers, three convolutional layers, and four convolutional layers.

[0031] Step 3, Construct the feature reconstruction model: The feature reconstruction part of the model reconstructs the multi-scale feature extraction results, optimizes the network model, and improves the classification accuracy. Among them, the feature reconstruction model includes three skip modules and one convolutional layer. Among them, each skip module includes four convolutional layers, four normalization operation layers, and four activation function layers.

[0032] Step 4, Determine the prediction model: Use the training set to train the network model. After the number of training times reaches the preset threshold, use the test set to test the trained model. When the accuracy of the network model stabilizes within a certain established range, it is considered that the network model training is completed, and the parameters of the network model are saved. After the network training is completed, fix the network model parameters and determine this network model as the final nodule prediction model. When it is necessary to predict the benign or malignant of thyroid nodule ultrasound images later, the image can be directly input into the network model to obtain the final prediction result.

[0033] Example 2

[0034] As Figure 1 shown, a multi-scale and multi-region thyroid nodule prediction method specifically includes the following steps:

[0035] Step 1: Prepare the training data. The original images provided by the hospital have a resolution of 1440×1080 pixels. The invalid information in this dataset is removed, the size is unified, and operations such as image normalization are performed. The resolution of the processed images is 640×640 pixels. Then, data augmentation is performed by methods such as increasing brightness and contrast, flipping, and rotating to obtain the final dataset, which consists of 3516 ultrasound images in JPG format. This dataset is divided into a training set and a test set at a ratio of 7:3.

[0036] Step 2: Construct a feature extraction model. The structure diagram of the feature extraction model is as shown in Figure 2 the feature extraction part. The feature extraction part of the model includes a high-frequency and low-frequency feature extraction module and a multi-scale feature extraction module. The input thyroid ultrasound images are respectively input into three thyroid nodule images of different scales into the high-frequency and low-frequency feature extraction module according to the three aspects of the nodule contour, nodule shape, and nodule edge in the morphological features of the nodule ultrasound image. Their sizes are 640×640, 320×320, and 160×160 respectively; the output results are sent to the multi-scale feature extraction module for further feature extraction. This part performs feature extraction on images of different scales;

[0037] Among them, the high-frequency and low-frequency feature extraction module includes a high-frequency module and a low-frequency module, as shown in Figure 3 The high-frequency module includes convolution operations and pooling operations, and the low-frequency module includes average pooling operations, upsampling operations, and convolution operations. Among them, the convolution kernel size in the convolution operation is 2×2. The input image is divided into high-frequency X H and low-frequency X L , and the resolution of the low-frequency image is reduced to half of the high-frequency image. The output result of the high-frequency module is Y H , and the output result of the low-frequency module is Y L . The final output result can be expressed as:

[0038] Y = Y H + Y L

[0039] Among them, the expressions of Y H and Y L are:

[0040] Y H = f(X H ; W H→H ) + upsample[f(X L ; W L→H ), 2]

[0041] Y L = f(X L ; W L→L ) + f[pool(X H , 2); WH→L

[0042] Among them: upsample represents the upsampling operation, pool represents the pooling operation, and f(X,W) represents the convolution operation with convolution kernel W.

[0043] Such as Figure 4 As shown, the multi-scale feature extraction module includes four convolutional blocks and a short-circuit mechanism. Among them, the short-circuit mechanism is that the input is directly connected to the output; the four convolutional blocks respectively include convolutional layer 1, convolutional layer 2, convolutional layer 3, and convolutional layer 4. The convolutional kernel size of convolutional layer 1 is 3×3, and the dilation rate is 1; convolutional layer 2 includes a 3×3 convolutional kernel with a dilation rate of 3 and a 1×1 convolutional kernel with a dilation rate of 1; convolutional layer 3 includes a 3×3 convolutional kernel with a dilation rate of 1, a 3×3 convolutional kernel with a dilation rate of 3, and a 1×1 convolutional kernel with a dilation rate of 1; convolutional layer 4 includes a 3×3 convolutional kernel with a dilation rate of 1, a 3×3 convolutional kernel with a dilation rate of 3, a 3×3 convolutional kernel with a dilation rate of 7, and a 1×1 convolutional kernel with a dilation rate of 1;

[0044] Step 3, construct a feature reconstruction model: The feature reconstruction part of the model performs feature reconstruction on the multi-scale feature extraction results; as Figure 5 shown, the feature reconstruction model includes three skip modules and a convolutional layer, where the convolutional kernel size of one convolutional layer is 1×1;

[0045] Among them, the skip module is as Figure 6 shown, and includes four convolutional layers, four normalization operation layers, and four activation function layers. Among them, the convolutional kernel sizes of the convolutional layers are 1×1 and 3×3, the stride is 1 for both, and the rectified linear unit is used as the activation function.

[0046] The rectified linear unit is defined as follows:

[0047]

[0048] Step 4, determine the prediction model. During the network training process, the number of training iterations is set to 500 respectively, and the learning rate is set to 10 -3 , and when the prediction accuracy rate reaches more than 95%, it is considered that the network training is completed. After the network training is completed, the network model parameters are fixed. The network model using these network model parameters is determined as the final prediction model.

[0049] Analyze the prediction effect of the multi-scale multi-region thyroid nodule prediction model: Input the test set divided in step 1 into the prediction model, and use accuracy, recall rate, and precision as evaluation indicators. As shown in the experimental results of the network model in Table 1, it can be seen that the network model proposed by the present invention has the best effect.

[0050] ​Table 1 Experimental Results of the Network Model

[0051]

[0052]

[0053] Among them, the implementations of convolution, activation function, pooling, and upsampling are algorithms well-known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical documents.

[0054] The preferred embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solutions of the present invention, and these simple modifications all fall within the protection scope of the present invention.

[0055] In addition, it should be noted that, among the various specific technical features described in the above specific embodiments, they can be combined in any appropriate manner without contradiction. To avoid unnecessary repetition, the present invention will not separately describe various possible combination methods.

[0056] Furthermore, any combination can be made among the various different embodiments of the present invention, as long as it does not violate the idea of the present invention, and it should also be regarded as the content disclosed by the present invention.

Claims

1. A multi-scale and multi-region prediction method for thyroid nodules, characterized in that, The specific steps are as follows: Step 1: Prepare the training dataset: Collect the thyroid ultrasound image set of thyroid patients, perform image preprocessing, and construct the training dataset; Step 2: Construct the feature extraction model: The feature extraction model includes a high-frequency and low-frequency feature extraction module and a multi-scale feature extraction module; Input thyroid nodule images of different scales and different regions into the high-frequency and low-frequency feature extraction module, extract features from three aspects: nodule contour, nodule shape, and nodule edge, and use the output result as the input of the multi-scale feature extraction module for further feature extraction; Step 3: Construct the feature reconstruction model: The feature reconstruction model performs feature reconstruction on the multi-scale feature extraction result obtained in Step 2, optimizes the network model, and improves the classification accuracy; Step 4: Determine the prediction model: Use the training set to train the network model. After the number of training times reaches the preset threshold, use the test set to test the trained model. When the accuracy of the network model stabilizes within a certain established range, it is considered that the network model training is completed, and the parameters of the network model are saved; After the network training is completed, fix the network model parameters, and determine this network model as the final nodule prediction model; Finally, based on the trained network model, predict the type of thyroid nodule; In Step 2, the high-frequency and low-frequency feature extraction module includes a high-frequency module and a low-frequency module. Among them, the high-frequency module is used to perform convolution and pooling operations on the data, and the low-frequency module is used to perform average pooling and upsampling operations on the data; The multi-scale feature extraction module includes four different convolutional blocks and a short-circuit mechanism; The four different convolutional blocks are a convolutional layer, two convolutional layers, three convolutional layers, and four convolutional layers respectively; In Step 3, the feature reconstruction model includes three skip modules and a convolutional layer; Among them, each skip module includes four convolutional layers, four normalization operation layers, and four activation function layers.

2. The multi-scale and multi-region thyroid nodule prediction method according to claim 1, characterized in that In Step 1, the image set refers to the thyroid nodule ultrasound images that have been confirmed as benign or malignant by puncture pathology during thyroid ultrasound examinations in the hospital; The image preprocessing includes removing invalid information, unifying the size, image normalization, and ultrasonic contrast enhancement, and dividing the training dataset into a training set and a test set according to a ratio of 7:3.

Images