Method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model, segmentation and classification method, and computer equipment

Abstract

The invention relates to a method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model, a segmentation and classification method, and a computer device, which belong to the field of image processing technology and solve the problems of multi-scale thyroid nodules, blurred nodule edges, imbalanced benign and malignant classification, and data imbalance. The method includes: using FCN as the backbone shared network, a UNet decoding layer as the segmentation branch network, and ResNet34 as the classification branch network. The backbone shared network uses parameter hard sharing to extract shallow features from the input ultrasound image, and shares the extracted features to the two branch networks. In the segmentation branch network, a deep convolution block is first introduced after the backbone shared network to obtain the deep features of the segmentation branch. Secondly, the shallow features obtained in each feature extraction stage are subjected to a jump connection operation with a multi-scale convolution attention module. M‑CBAM is combined before and after the residual module of the classification branch, and the classification performance is optimized by M‑CBAM and the residual module. The present invention is suitable for the segmentation and classification of ultrasound thyroid nodules.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to the segmentation and classification of ultrasonic thyroid nodules. Background Art

[0002] The incidence of thyroid cancer has been increasing in recent years, making early detection, diagnosis, and treatment crucial. Ultrasound examination offers advantages in differential diagnosis and assessment of benign and malignant thyroid tumors, such as safety, non-invasiveness, and high diagnostic accuracy. It has been widely used in clinical practice and is the preferred method for detecting thyroid nodules. The shape and aspect ratio of thyroid nodules are crucial features in diagnosing benign and malignant tumors, and the segmentation and classification of thyroid nodules are crucial for clinical treatment.

[0003] Deep convolutional neural networks (CNNs) can significantly improve the accuracy of object classification and segmentation and are widely used in biomedicine. They enable automated feature detection, overcoming the inherent difficulties of feature detection in traditional pattern analysis techniques. Convolutional neural networks, such as UNet and ResNet, have achieved impressive results in medical image segmentation and classification, respectively, and have been widely used in their respective fields. The attention mechanism enables the model to allocate important feature information across channels and spaces, significantly improving its efficiency and performance.

[0004] Deep learning-based thyroid nodule segmentation methods require no human intervention and significantly increase the speed of thyroid nodule segmentation. However, many models use a multi-stage segmentation approach, further increasing the computational load. Furthermore, these models do not adequately utilize multi-scale information extraction and attention mechanisms, and they rely heavily on large amounts of data.

[0005] The thyroid nodule classification method based on deep learning can greatly improve the classification accuracy, but the network training is highly dependent on the amount of data. The lack of benign and malignant data and the deepening of the network layers may cause class imbalance and overfitting, making it impossible to effectively identify benign and malignant thyroid nodules, resulting in misidentification and missed identification. Summary of the Invention

[0006] The purpose of the present invention is to solve the problems of multi-scale, fuzzy nodule edges, imbalanced benign and malignant classification, and data imbalance of existing thyroid nodules, and provide a method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model, a segmentation and classification method, and a computer device.

[0007] The present invention is achieved through the following technical solutions. On the one hand, the present invention provides a method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model. The joint ultrasound thyroid nodule segmentation and classification model uses FCN as the backbone network framework, and the branch networks are respectively a segmentation branch network based on the UNet decoding layer and a classification branch network based on ResNet34. The method includes:

[0008] The FCN performs convolution and pooling operations on the image to extract shallow features and obtain the output feature tensor of the FCN;

[0009] Inputting the output feature tensor of the FCN into the segmentation branch network and the classification branch network respectively;

[0010] The segmentation branch network is based on the UNet decoding layer, passes the output feature tensor of the FCN through a deep convolution block to extract deep features, then upsamples the deep features, and splices the upsampling result with the feature tensor of each feature extraction stage of the FCN after skip connection to obtain a spliced ​​feature tensor, and finally performs segmentation convolution processing on the spliced ​​feature tensor to obtain a segmentation result, wherein the skip connection is a skip connection based on the multi-scale convolutional attention module M-CBAM;

[0011] The classification branch network is based on ResNet34, and performs M-CBAM and residual module operations on the output feature tensor of the FCN to obtain the classification result;

[0012] Training the segmentation branch network and the classification branch network respectively to obtain a loss function of the segmentation branch network and a loss function of the classification branch network;

[0013] According to the loss function of the segmentation branch network and the loss function of the classification branch network, a total loss function is obtained; according to the total loss function, a preset number of iterations of training are performed to obtain the optimal combined ultrasound thyroid nodule segmentation and classification model to complete the model establishment.

[0014] Furthermore, the M-CBAM includes a channel attention module and a spatial attention module;

[0015] The channel attention module first sums the maximum pooling and average pooling through feature splicing, and then trains them through the multi-layer perceptron MLP to enhance the correlation between the two parts of the feature tensor. Then, after the Sigmoid activation function, the output feature tensor is bitwise multiplied with the input feature tensor to obtain the feature tensor output by the channel attention module;

[0016] The spatial attention module introduces a multi-scale convolution block MSCB to improve the model's ability to process multi-scale nodules. The output feature tensor is then bitwise multiplied with the input feature tensor through a Sigmoid activation function to obtain the feature tensor output by the spatial attention module.

[0017] Furthermore, the M-CBAM attention mechanism execution steps specifically include:

[0018] The feature tensor output by each feature extraction stage of the FCN is used as the input feature tensor, and the output feature tensor F1 is obtained through the channel attention module;

[0019] Passing the output feature tensor F1 through the spatial attention module to obtain an output feature tensor F2;

[0020] The output feature tensor F2 is multiplied by the feature tensor Fi bit by bit to obtain the feature tensor F3 output by the M-CBAM.

[0021] Furthermore, obtaining the output feature tensor F1 specifically includes:

[0022] The output feature tensor F1 is obtained by using the channel attention module expression, where the channel attention module expression is specifically:

[0023]

[0024] Among them, F1 is the output feature tensor obtained by the channel attention module, δ is the Sigmoid activation function, MLP is the multi-layer perceptron, MaxPool is the maximum pooling, AvgPool is the average pooling, and F is the input feature tensor. It is bitwise multiplication.

[0025] Furthermore, obtaining the output feature tensor F2 specifically includes:

[0026] The output feature tensor F2 is obtained by using the spatial attention module expression, where the spatial attention module expression is specifically:

[0027]

[0028] Among them, F2 is the output feature tensor obtained by the spatial attention module, δ is the Sigmoid activation function, MSCB is the multi-scale convolution block, MaxPool is the maximum pooling, AvgPool is the average pooling, and F1 is the output feature tensor obtained by the channel attention module. It is bitwise multiplication.

[0029] Furthermore, performing M-CBAM and residual module operations on the output feature tensor of the FCN to obtain a classification result specifically includes:

[0030] Pass the output feature tensor of the FCN through the M-CBAM to obtain an output feature tensor B1;

[0031] Pass the output feature tensor B1 through a plurality of the residual modules to obtain an output feature tensor B2;

[0032] The output feature tensor B2 is passed through the M-CBAM to obtain the output feature tensor B3.

[0033] Furthermore, the residual module includes a first convolutional layer and a second convolutional layer;

[0034] The first convolutional layer includes 3×3 convolution, batch normalization and sparse activation function;

[0035] The second convolutional layer includes 3×3 convolution and batch normalization processing;

[0036] After the output feature tensor B1 passes through the first convolution layer and the second convolution layer in sequence, a first convolution result is obtained;

[0037] After the output feature tensor B1 passes through 1×1 convolution and BN, a second convolution result is obtained;

[0038] The first convolution result and the second convolution result are added together, and the added result is passed through a ReLU activation function to obtain the output feature tensor of the residual module.

[0039] In a second aspect, the present invention provides a multi-task ultrasound-based thyroid nodule segmentation and classification method, in which the thyroid nodule ultrasound image is input into the multi-task ultrasound-based thyroid nodule segmentation and classification model to obtain the ultrasound thyroid nodule segmentation and classification results.

[0040] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, a method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model as described above is executed.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein a plurality of computer instructions are stored in the computer-readable storage medium, and the plurality of computer instructions are used to enable a computer to execute a method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model as described above.

[0042] Beneficial effects of the present invention:

[0043] First, the method of the present invention uses (Fully Convolutional Networks, FCN) as the backbone shared network, the UNet decoding layer as the segmentation branch network and the ResNet34 as the classification branch network. The backbone shared network uses parameter hard sharing to extract shallow features from the input ultrasound image, and shares the extracted features with the two branch networks. In the segmentation branch network, the deep layer convolutional block (DLCB) is first introduced after the backbone shared network to obtain the deep features of the segmentation branch. Secondly, the shallow features obtained in the first feature extraction stage Conv1, the second feature extraction stage Conv2, the third feature extraction stage Conv3, and the fourth feature extraction stage Conv4 are maintained through a jump connection operation with a multiscale convolutional block attention module (M-CBAM) to maintain the segmentation edge feature contours of the above four stages, thereby reducing the problem of blurred nodule edges. M-CBAM is combined before and after the classification branch residual module, and the classification performance is optimized by M-CBAM and the residual module.

[0044] Secondly, the present invention integrates M-CBAM into the two branch networks and adds deep convolution blocks to the segmentation branch network, which solves the data imbalance problem and can make more accurate predictions when the amount of data is small.

[0045] Through comparative experiments, it is proved that the method proposed in the present invention has better segmentation and classification performance than the deep learning network for separate segmentation and classification.

[0046] Multi-task methods can simultaneously learn multiple related tasks and share feature information across all tasks, thereby improving the performance of each task. Thyroid nodule segmentation and classification require a large number of common nodule features. Therefore, combining ultrasound thyroid nodule segmentation and classification with a multi-task approach to achieve both segmentation and classification of thyroid nodule ultrasound images is of great research significance.

[0047] The present invention is applicable to the segmentation and classification of ultrasonic thyroid nodules. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1Schematic diagram of the multi-task ultrasound thyroid nodule segmentation and classification model of the present invention;

[0050] Figure 2 The multi-scale convolutional attention module network structure of the present invention;

[0051] Figure 3 This is a schematic diagram of the ResNet34 network structure of the present invention;

[0052] Figure 4 Comparison of segmentation results of different networks;

[0053] Figure 5 Radar charts for different network segmentation indicators;

[0054] Figure 6 Cumulative distribution curves of the indicators obtained for each network;

[0055] Figure 7 ROC curves for different classification algorithms used on the test data set. DETAILED DESCRIPTION

[0056] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but are not to be construed as limiting the present invention.

[0057] Embodiment 1: A method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model. The joint ultrasound thyroid nodule segmentation and classification model uses FCN as the backbone network framework, and the branch networks are a segmentation branch network based on the UNet decoding layer and a classification branch network based on ResNet34. The method includes:

[0058] The FCN performs convolution and pooling operations on the image to extract shallow features and obtain the output feature tensor of the FCN;

[0059] Inputting the output feature tensor of the FCN into the segmentation branch network and the classification branch network respectively;

[0060] The segmentation branch network is based on the UNet decoding layer, passes the output feature tensor of the FCN through a deep convolution block to extract deep features, then upsamples the deep features, and splices the upsampling result with the feature tensor of each feature extraction stage of the FCN after skip connection to obtain a spliced ​​feature tensor, and finally performs segmentation convolution processing on the spliced ​​feature tensor to obtain a segmentation result, wherein the skip connection is a skip connection based on the multi-scale convolutional attention module M-CBAM;

[0061] The classification branch network is based on ResNet34, and performs M-CBAM and residual module operations on the output feature tensor of the FCN to obtain the classification result;

[0062] Training the segmentation branch network and the classification branch network respectively to obtain a loss function of the segmentation branch network and a loss function of the classification branch network;

[0063] According to the loss function of the segmentation branch network and the loss function of the classification branch network, a total loss function is obtained; according to the total loss function, a preset number of iterations of training are performed to obtain the optimal combined ultrasound thyroid nodule segmentation and classification model to complete the model establishment.

[0064] It should be noted that the loss functions of the segmentation branch network and the classification branch network both use the cross-entropy loss function. The segmentation and classification networks are trained simultaneously, and the backbone network is identical, so there are a large number of shared parameters in the backbone network. For example, the input image size, batch size, learning rate, number of iterations, and the size and number of convolution kernels in the backbone network are all shared parameters. The more shared parameters, the lower the probability of overfitting.

[0065] The segmentation and classification loss functions are weighted and summed so that the impact of the segmentation and classification losses on the shared layer is roughly the same, and the total segmentation and classification loss after weighted summation is obtained. By setting the number of iterations, training is performed for a certain number of iterations. After one training is completed, the result model is saved, and the next training is started. The results of each training are compared and the optimal model (the combined ultrasound thyroid nodule segmentation and classification model) is saved.

[0066] If the average loss of the last batch in each training round on the training set does not decrease over five consecutive rounds, the learning rate is adjusted to 0.9 times the previous value. By mutually adjusting shared parameters between segmentation and classification, such as updating the learning rate, to optimize shared parameters, the ability of both to learn features is enhanced, reducing their respective losses and, consequently, the total loss.

[0067] In this embodiment, first, the method of this embodiment uses (Fully Convolutional Networks, FCN) as the backbone shared network, the UNet decoding layer as the segmentation branch network, and the ResNet34 as the classification branch network. The backbone shared network uses parameter hard sharing to extract shallow features from the input ultrasound image, and shares the extracted features with the two branch networks. In the segmentation branch network, a deep layer convolutional block (DLCB) is first introduced after the backbone shared network to obtain the deep features of the segmentation branch. Secondly, the shallow features obtained in each feature extraction stage of FCN are maintained through a jump connection operation with a multiscale convolutional block attention module (M-CBAM) to reduce the blurring problem of nodule edges.

[0068] M-CBAM is combined before and after the classification branch residual module, and the classification performance is optimized through M-CBAM and the residual module.

[0069] Secondly, this implementation integrates M-CBAM into the two branch networks and adds deep convolution blocks to the segmentation branch network, which solves the data imbalance problem and can make more accurate predictions when the amount of data is small.

[0070] Finally, this embodiment uses the loss functions of the two branch networks for joint training to make the shared parameters reach the optimal value, which can enhance the ability of both to learn features, reduce their respective losses, and thus reduce the total loss, thereby improving the model's recognition accuracy for multi-scale thyroid nodules.

[0071] Embodiment 2: This embodiment further defines the method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model described in embodiment 1. In this embodiment, the M-CBAM is further defined, specifically including:

[0072] The M-CBAM includes a channel attention module and a spatial attention module;

[0073] The channel attention module first sums the maximum pooling and average pooling through feature splicing, and then trains them through the multi-layer perceptron MLP to enhance the correlation between the two parts of the feature tensor. Then, after the Sigmoid activation function, the output feature tensor is bitwise multiplied with the input feature tensor to obtain the feature tensor output by the channel attention module;

[0074] The spatial attention module introduces a multi-scale convolution block MSCB to improve the model's ability to process multi-scale nodules. The output feature tensor is then bitwise multiplied with the input feature tensor through a Sigmoid activation function to obtain the feature tensor output by the spatial attention module.

[0075] In this embodiment, in the M-CBAM channel attention module, in order to enable the fully connected layer (FC) of the multilayer perceptron (MLP) to simultaneously calculate the features of the maximum pooling (MaxPool) and average pooling (AvgPool);

[0076] In the M-CBAM spatial attention module, in order to reduce the problem of variable scale of thyroid nodule morphology, the original 7×7 convolution kernel of CBAM is replaced with a multi-scale convolutional block (MSCB).

[0077] This embodiment can reduce the problems of image edge information blurring, information loss, and variable morphological scale of thyroid nodules.

[0078] Embodiment 3: This embodiment further limits the method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model described in Embodiment 2. In this embodiment, the execution steps of the attention mechanism of the M-CBAM are further limited, specifically including:

[0079] The M-CBAM attention mechanism execution steps specifically include:

[0080] The feature tensor output by each feature extraction stage of the FCN is used as the input feature tensor, and the output feature tensor F1 is obtained through the channel attention module;

[0081] Passing the output feature tensor F1 through the spatial attention module to obtain an output feature tensor F2;

[0082] The output feature tensor F2 is bitwise multiplied by the feature tensor Fi to obtain the feature tensor F3 output by the M-CBAM.

[0083] This implementation provides a specific method for executing the attention mechanism steps of M-CBAM.

[0084] In this implementation, the number of feature extraction stages of FCN can be set to 5 to avoid excessive data volume and deep structure.

[0085] Embodiment 4: This embodiment further limits the method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model described in embodiment 3. In this embodiment, the output feature tensor F1 is further limited, specifically including:

[0086] The output feature tensor F1 is obtained by using the channel attention module expression, where the channel attention module expression is specifically:

[0087]

[0088] Among them, F1 is the output feature tensor obtained by the channel attention module, δ is the Sigmoid activation function, MLP is the multi-layer perceptron, MaxPool is the maximum pooling, AvgPool is the average pooling, and F is the input feature tensor. It is bitwise multiplication.

[0089] This embodiment provides a method for obtaining the output feature tensor F1.

[0090] Embodiment 5: This embodiment further limits the method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model described in embodiment 3. In this embodiment, the output feature tensor F2 is further limited, specifically including:

[0091] The output feature tensor F2 is obtained by using the spatial attention module expression, where the spatial attention module expression is specifically:

[0092]

[0093] Among them, F2 is the output feature tensor obtained by the spatial attention module, δ is the Sigmoid activation function, MSCB is the multi-scale convolution block, MaxPool is the maximum pooling, AvgPool is the average pooling, and F1 is the output feature tensor obtained by the channel attention module. It is bitwise multiplication.

[0094] This embodiment provides a method for obtaining the output feature tensor F2.

[0095] Embodiment 6: This embodiment further limits the method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model described in embodiment 1. In this embodiment, the M-CBAM and residual module operations are performed on the output feature tensor of the FCN to obtain the classification result, which is further limited, specifically including:

[0096] Specifically include:

[0097] Pass the output feature tensor of the FCN through the M-CBAM to obtain an output feature tensor B1;

[0098] Pass the output feature tensor B1 through a plurality of the residual modules to obtain an output feature tensor B2;

[0099] The output feature tensor B2 is passed through the M-CBAM to obtain the output feature tensor B3.

[0100] In this embodiment, the output feature tensor B3 is obtained by two M-CBAMs and a residual structure, wherein the first M-CBAM is used to extract key information from the input feature tensor B, reduce the influence of useless shallow feature information, and thereby improve the effective feature extraction capability of the residual module in the residual structure, and obtain the output feature tensor B1. The residual module in the residual structure is used to extract the deep features of B1 in the classification branch on the basis of obtaining effective key information, increase the utilization rate of the classification deep feature information, and obtain the output feature tensor B2. The second M-CBAM extracts key information from B2 and reduces the influence of useless deep feature information. The network structure of the residual module is as follows: Figure 3 shown.

[0101] Adding M-CBAM before and after the ResNet34 residual network structure to improve the ability to extract classification information can effectively solve the problems of gradient explosion, gradient disappearance, and possible network degradation as the network depth increases.

[0102] Embodiment 7: This embodiment further limits the method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model described in embodiment 6. In this embodiment, the residual module is further limited, specifically including:

[0103] The residual module includes a first convolutional layer and a second convolutional layer;

[0104] The first convolutional layer includes 3×3 convolution, batch normalization and sparse activation function;

[0105] The second convolutional layer includes 3×3 convolution and batch normalization processing;

[0106] After the output feature tensor B1 passes through the first convolution layer and the second convolution layer in sequence, a first convolution result is obtained;

[0107] After the output feature tensor B1 passes through 1×1 convolution and BN, a second convolution result is obtained;

[0108] The first convolution result and the second convolution result are added together, and the added result is passed through a ReLU activation function to obtain the output feature tensor of the residual module.

[0109] In this embodiment, the final addition enables the residual module to protect the integrity of information by bypassing the input information to the output.

[0110] The residual module in this embodiment constitutes the residual network in the ResNet34 network, which can prevent the problems of gradient explosion and gradient disappearance.

[0111] Embodiment 8: This embodiment is a method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model described in embodiment 1, and specifically includes:

[0112] The thyroid nodule ultrasound image is input into the multi-task ultrasound thyroid nodule segmentation and classification model to obtain the ultrasound thyroid nodule segmentation and classification results.

[0113] The method in this embodiment can address the problems of multi-scale thyroid nodules, fuzzy nodule edges, and imbalanced benign and malignant classification, and obtain high-precision ultrasonic thyroid nodule segmentation and classification results.

[0114] Embodiment 9: This embodiment is a specific example of a method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model as described above, specifically:

[0115] To address the issues of multi-scale thyroid nodules, blurred nodule edges, and imbalanced benign and malignant classification in ultrasound images, this embodiment proposes a multi-task ultrasound thyroid nodule segmentation and classification model. This model uses the FCN as the backbone network framework, with branch networks consisting of a segmentation branch based on the UNet decoding layer and a classification branch based on ResNet34. The UNet decoding layer includes a decoding block, an improved multi-scale convolutional attention module (M-CBAM) (Convolutional Block Attention Module, CBAM), and a deep convolutional block (DLCB). ResNet34 has M-CBAM before and after the residual module.

[0116] Among them, the multi-scale convolutional attention module M-CBAM of the improved CBAM is introduced in the skip connection of the UNet decoding layer.

[0117] In the M-CBAM channel attention module, in order to enable the fully connected layer (FC) layer of the multilayer perceptron (MLP) to simultaneously calculate the features of the maximum pooling (MaxPool) and average pooling (AvgPool), the maximum pooling and average pooling are first summed through feature splicing, and then trained by the multilayer perceptron (MLP) to enhance the correlation between the two parts of the feature tensors. After passing the Sigmoid activation function, the output feature tensor is bitwise multiplied with the input feature tensor to obtain the feature tensor output by the channel attention module.

[0118] In the M-CBAM spatial attention module, in order to reduce the problem of variable morphological scale of thyroid nodules, the original 7×7 convolution kernel of CBAM is replaced with a multi-scale convolutional block (MSCB). By introducing the multi-scale convolutional block MSCB, the model's ability to process multi-scale nodules is improved. The output feature tensor obtained after the Sigmoid activation function is bitwise multiplied with the input feature tensor to obtain the feature tensor output by the spatial attention module.

[0119] The execution steps of the M-CBAM attention mechanism are as follows. Taking the first feature extraction stage Conv1 as an example, the output feature tensor (shallow feature) obtained by Conv1 is used as the input feature tensor of the multi-scale convolutional attention module M-CBAM, named F. The output feature tensor F1 is obtained through the channel attention module. Formula (1) gives the expression of the channel attention module.

[0120]

[0121] Where F1 is the output feature tensor obtained by the channel attention module, δ is the Sigmoid activation function, MLP is the multi-layer perceptron, MaxPool is the maximum pooling, AvgPool is the average pooling, and F is the input feature tensor. It is bitwise multiplication.

[0122] The output feature tensor F1 is passed through the spatial attention module to obtain the output feature tensor F2. Formula (2) gives the expression of the spatial attention module.

[0123]

[0124] Among them, F2 is the output feature tensor obtained by the spatial attention module, δ is the Sigmoid activation function, MSCB is the multi-scale convolution block, MaxPool is the maximum pooling, AvgPool is the average pooling, and F1 is the output feature tensor obtained by the channel attention module. The output feature tensor F2 obtained by the spatial attention module and the input feature tensor F are bitwise multiplied to obtain the final output feature tensor F3 improved by spatial attention and channel attention. The output feature tensor obtained by the channel and spatial attention modules can not only keep the boundary edge information from being lost, but also improve the utilization rate of feature information and improve the ability to segment multi-scale nodules. The network structure of the M-CBAM multi-scale convolutional attention module is as follows: Figure 2 shown.

[0125] Finally, the high-level feature map upsampled by the decoder is spliced ​​together, and the final segmentation result is obtained after two 3×3 convolutions and one 1×1 convolution, a total of four upsamplings, as follows:

[0126] First, upsample the deep features to obtain the first upsampling result;

[0127] The first upsampling result is concatenated with the shallow features of conv4 through the feature tensor output by M-CBAM to obtain the first concatenation result;

[0128] Upsampling the first splicing result to obtain a second upsampling result;

[0129] The second upsampling result is concatenated with the shallow features of conv3 through the feature tensor output by M-CBAM to obtain the second concatenation result;

[0130] By analogy, the fourth concatenation result is obtained by finally concatenating the shallow features of conv1 with the feature tensor output by M-CBAM.

[0131] The fourth splicing result is then subjected to two 3×3 convolutions and one 1×1 convolution to obtain the final segmentation result.

[0132] Among them, the last two 3×3 convolutions are consistent with the operations of other layers, and the convolution operation of feature extraction is continued. The size of the extracted image remains unchanged, and the number of channels becomes 32. The last 1×1 is to convert the number of channels of 32 to the output channel number 2, that is, the foreground (target area) and background. Finally, softmax obtains the maximum probability value to obtain the foreground segmentation result.

[0133] There is no limit to the number of stages, but as the number of stages increases, the number of parameters will also increase significantly, and the more stages the better the effect is not necessarily the case. Here, 5 layers are defined. The advantage is that by extracting feature information from 5 stages, the feature extraction capability can be greatly improved, including deep features and shallow features.

[0134] In the ResNet34 network as the classification branch, M-CBAM is added before and after the ResNet34 residual network structure to improve the ability to extract classification information and solve problems such as gradient explosion and gradient disappearance that may occur as the network depth deepens, and network degradation may occur.

[0135] First, a multi-scale convolutional attention module M-CBAM is added before the residual structure. The shallow features of the output feature tensor obtained in the fifth feature extraction stage Conv5 are used as the input feature tensor of the multi-scale convolutional attention module M-CBAM in the classification branch. The input feature tensor is named B, and the output feature tensor obtained by the multi-scale convolutional attention module M-CBAM is named B1.

[0136] Secondly, the residual network structure in the ResNet34 network is used to prevent gradient explosion and vanishing problems. Taking a residual module as an example, two convolutional layers are performed on B1 to extract deep features of B1 in the classification branch and increase the utilization of classification feature information. The first convolutional layer includes 3×3 convolution, batch normalization (BN), and sparse activation function (RectiBied Liner Uints, ReLU). The second convolutional layer also includes 3×3 convolution and batch normalization.

[0137] After B1 passes through two convolutional layers, it is added to B1 through 1×1 convolution and BN. After the addition, a ReLU activation function is used to obtain the output feature tensor of a residual module. By bypassing the input information to the output, the integrity of the information can be protected.

[0138] After passing through all the residual modules, a multi-scale convolutional attention module M-CBAM is added after the residual structure. The output feature tensor B3 is obtained through two M-CBAMs and the residual structure. The first M-CBAM is used to extract the key information in the input feature tensor B, reduce the influence of useless shallow feature information, and thus improve the effective feature extraction ability of the residual module in the residual structure. The residual module in the residual structure is used to extract the deep features of B1 in the classification branch on the basis of obtaining effective key information, increase the utilization rate of the deep feature information of the classification, and obtain the output feature tensor B2. The second M-CBAM extracts the key information in B2 and reduces the influence of useless deep feature information. The network structure of the residual module is as follows: Figure 3 shown.

[0139] Embodiment 10: This embodiment is based on a specific embodiment of the multi-task ultrasound thyroid nodule segmentation and classification method as described above, specifically:

[0140] First, FCN is used to perform convolution and pooling operations on the image to extract shallow features, and then the output feature tensor of FCN is used as the input of the segmentation and classification branch network.

[0141] Secondly, the segmentation branch network first passes the input feature tensor through a deep convolution block to extract deep features, and then performs an upsampling decoding operation with an M-CBAM jump connection.

[0142] The classification branch network simultaneously performs M-CBAM and residual module operations on the FCN input feature tensor, and finally obtains the segmentation and classification results.

[0143] The overall framework of the model is as follows Figure 1 shown.

[0144] Below, the technical effects of the present invention are given in combination with experiments and analysis of data.

[0145] 1. Experimental data and parameter settings

[0146] The experimental data for this paper was provided by Shanghai Chest Hospital, consisting of 450 ultrasound images of thyroid nodules and their outlines annotated by experts. Of these, 304 ultrasound images of thyroid nodules were classified as either benign or malignant, with 73 cases being benign and 231 being malignant. Therefore, these 304 ultrasound images with image sample labels and categories were selected as the existing dataset. After removing privacy information, the images were uniformly cropped to a 512×512 size. The training set data was augmented ninefold using rotation, mirroring, and scaling operations. After augmentation, the training set contained 1,647 data samples. The benign and malignant data were randomly divided and combined into a training set, a validation set, and a test set, respectively, in a 6:2:2 ratio.

[0147] The specific parameters of the network in the experiment are as follows: the input image size is 512×512×1, the batch size is 16, the initial learning rate is set to 0.0001, the model optimizer is Adaptive Moment Estimation (Adam), and both the segmentation and classification branch networks use cross entropy loss as the loss function. The segmentation cross entropy loss (SegCELoss) and the classification cross entropy loss (ClaCELoss) are summed to obtain the joint cross entropy loss function (JCELoss). α and β are each set to 0.5. Formula (3) is shown below.

[0148] JCELoss=αSegCELoss+βClaCELoss (3)

[0149] The maximum number of iterations is 100. During training, if the loss on the validation set does not decrease after 10 consecutive epochs, the learning rate is reduced to 0.9 times the original value. The experimental hardware environment consists of an Intel(R) Xeon(R) 4216 with a main frequency of 2.10GHz and two GeForce RTX 3070 graphics cards; the operating system is Ubuntu 20.04, and the programming language is Python 3.8.5. All programs are implemented in the PyTorch framework.

[0150] 2. Evaluation Indicators

[0151] In terms of segmentation accuracy, the proposed method was compared with UNet, Attention-UNet, UNet++, Deeplabv3+, SegNet, and Multi-task Segmentation Network (MSNet). Evaluation metrics included the Dice Similarity Coefficient (DSC), Intersection of Union (IOU), Positive Predictive Value (PPV), and True Positive Rate (TPR).

[0152] In terms of classification accuracy, the proposed method was compared with ResNet34, ResNet50, ResNet101, DenseNet121, DenseNet169, and a multi-task classification network (MCNet). Evaluation metrics included accuracy (ACC), precision (PPV), recall (TPR), and receiver operating characteristic (ROC) and area under the curve (AUC).

[0153] TP, TN, FP, and FN represent the number of true positive, true negative, false positive, and false negative pixels in the classification results, respectively. The ROC curve is plotted with the true positive rate (SENS) as the ordinate and the false positive rate (1 – SPEC) as the abscissa. AUC is the area under the ROC curve. A value closer to 1 indicates better classification algorithm performance.

[0154] 3. Results and Analysis

[0155] 1) Segmentation result analysis

[0156] Figure 4 The segmentation results of different networks are given, where Figure 4 (a) is the original data, Figure 4 (b) is the gold standard of expert annotation, Figure 4 (c) to 4 (h) are the segmentation results of UNet, Attention-UNet, UNet++, Deeplabv3+, SegNet, MSNet, and the method of the present invention, JMNet. The wireframes in the figure mark some of the differences between the expert gold standard, the comparison network, and the segmentation results of the network of the present invention. The segmentation results are divided into 8 groups from top to bottom. The thyroid nodules in groups 1 to 3 are benign nodules, and those in groups 4 to 8 are malignant nodules. The malignant nodules in group 6 are spliced ​​from two malignant nodule pictures. Figure 4Judging from the segmentation results, the segmentation results of UNet, Attention-UNet, UNet++, and SegNet in groups 1, 3, and 8 show a lot of obvious over-segmentation. They mistakenly classify areas farther away from the expert-annotated outline as ROI areas and include them in the predicted results, or classify farther away non-ROI areas as ROI areas and display them outside the outline. Deeplabv3+ has slight over-segmentation, but its segmentation effect is much better than that of other comparison networks. The comparison networks in groups 2 and 4 generally have some over-segmentation. UNet, Attention-UNet, UNet++, and MSNet in groups 5 and 6 have obvious under-segmentation. The comparison networks in group 7 all have under-segmentation, and UNet and Attention-UNet have both under-segmentation and slight over-segmentation.

[0157] The comparison network has a better segmentation effect on small nodules, but there will be some under-segmentation in the segmentation of medium and large nodules. Figure 4 (i) The proposed network, using the M-CBAM multi-scale convolutional block, performs well in segmenting thyroid nodules of various shapes and sizes. The edge contours of the segmentation results remain close to the expert gold standard, and there are no significant errors in shape or aspect ratio. The results indicate that the proposed method performs best.

[0158] Seven network models, including the network of the present invention, were evaluated on ultrasound images of benign, malignant, and labeled thyroid nodules. Table 1 shows the segmentation results of the model of the present invention and the comparison model on the test set. The segmentation results are shown in Table 1. From left to right, they are DSC, IOU, PPV, and TPR. From top to bottom, they are UNet, Attention-UNet, UNet++, Deeplabv3+, SegNet, MSNet, and JMNet. The results show that when the data set is small, networks such as UNet and Attention-UNet have relatively similar segmentation capabilities, while the network of the present invention has the best segmentation performance index.

[0159] Table 1 Comparison of the proposed model with other network models in segmentation

[0160] Network DSC IOU PPV TPR UNet 0.5929±0.0118 0.4624±0.0101 0.6546±0.0327 0.6856±0.0503 Attention-UNet 0.5826±0.0137 0.4563±0.0110 0.6477±0.0438 0.6739±0.0941 UNet++ 0.5769±0.0159 0.4488±0.0170 0.6506±0.0558 0.6794±0.0600 DeepLabv3+ 0.5732±0.0110 0.4396±0.0136 0.5883±0.0177 0.6621±0.0150 SegNet 0.5985±0.0203 0.4711±0.0228 0.6367±0.0390 0.6982±0.0246 MSNet 0.6179±0.0168 0.4937±0.0146 0.6938±0.0239 0.6774±0.0425 JMNet 0.6428±0.0037 0.5196±0.0032 0.6832±0.0125 0.7192±0.0026

[0161] The proposed method utilizes a shared backbone FCN network to learn shallow features and connects the segmentation and classification branches to share these shallow features. The M-CBAM attention module preserves feature edges, mitigates scale polymorphism, and suppresses useless information and noise to a certain extent. The shared information, unlike that extracted by the individual networks, effectively improves both segmentation and classification performance. Figure 5A radar chart of seven network segmentation results is provided. In the chart, the loops displayed by the network of the present invention are basically all on the periphery, indicating that all indicators have good performance.

[0162] Figure 6 The cumulative distribution curves of DSC and IoU obtained by testing seven networks on all samples are shown. JMNet has the largest area under its cumulative distribution curve, has the fewest samples with a DSC of 0, and maintains a smoother curve than the other networks. The cumulative distribution curve of IoU also shows that JMNet achieves the best segmentation performance, demonstrating that the improved segmentation performance of multi-task networks with M-CBAM is reflected in the majority of test samples.

[0163] Table 2 shows the results of a five-fold cross-validation test on the test set (using DSC as an example). As can be seen from Table 2, the JMNet network of the present invention outperforms other segmentation models in every experimental evaluation metric. These results demonstrate that the multi-task network model JMNet can effectively improve thyroid nodule segmentation.

[0164] Table 2 Five-fold cross validation (DSC)

[0165] Network Fold1 Fold2 Fold3 Fold4 Fold5 UNet 0.5846 0.5762 0.5917 0.6074 0.6044 Attention-UNet 0.5864 0.5640 0.5694 0.5987 0.5944 UNet++ 0.5518 0.5978 0.5819 0.5670 0.5858 DeepLabv3+ 0.5798 0.5759 0.5631 0.5584 0.5887 SegNet 0.5641 0.5908 0.6145 0.5776 0.6296 MSNet 0.6124 0.5974 0.6119 0.6422 0.6255 JMNet 0.6402 0.6408 0.6406 0.6436 0.6490

[0166] Table 3 shows the segmentation results of the ablation experiment on the test set of the model of the present invention. As can be seen from Table 3, the present invention conducts ablation experiments on the basis of the multi-task network to improve the performance of segmentation. When the model adds the M-CBAM attention module at the jump connection, the indicators DSC, IOU and TPR are improved by about 5%, and PPV is improved by nearly 10%. When the model adds a deep convolution block after the backbone shared convolution to extract deep features, the indicators DSC and IOU are improved by about 11%, TPR is improved by about 3%, and PPV is improved by nearly 18%. When both the M-CBAM attention module and the deep convolution block are added, the indicators DSC and IOU are improved by about 14%, TPR is improved by about 9%, and PPV is improved by nearly 18%. After the ablation experiment, the network of the present invention surpasses the single-task segmentation network in segmentation performance indicators.

[0167] Table 3 Segmentation ablation experiments

[0168] M-CBAM DLCB DSC IOU PPV TPR 0.5046 0.3759 0.5224 0.6000 √ 0.5553 0.4226 0.6129 0.6333 √ 0.6114 0.4938 0.7007 0.6334 √ √ 0.6402 0.5276 0.7037 0.6879

[0169] 2) Classification results analysis

[0170] By testing different classification networks, the ROC curves of different networks are obtained, which are ResNet34, ResNet50, ResNet101, Densenet121, DenseNet169, MCNet and the ROC curve of the method JMNet of the present invention. The ROC curve results are as follows: Figure 7 As shown. Figure 7 It can be seen that the AUC areas of the classification networks are 0.565, 0.619, 0.587, 0.612, 0.663, 0.601, and 0.757, respectively. The network model of the present invention uses a multi-task approach to classify thyroid nodule ultrasound images with a small dataset and obtains an AUC area higher than that of other classification networks.

[0171] Table 4 shows the classification results of the model of the present invention and other classification networks on the test set. As can be seen from Table 4, in the ACC, AUC, PPV and TPR classification indicators, the performance indicators of other classification networks are not as good as the model of the present invention. This may be because when the data set is scarce, other networks cannot fully learn features. The ACC obtained by the network model of the present invention is improved by about 10% to 20% compared with other models, the AUC is improved by about 10% to 20%, the PPV is improved by about 10% to 15%, and the TPR is improved by about 10% to 18%. By comparing with other classification networks, it can be reflected that the method proposed by the present invention can better perform a more accurate cancer risk assessment on thyroid nodule ultrasound images. It alleviates the problem of data category imbalance to a certain extent and improves the classification performance. The results show that when the data set is small, the network of the present invention has the best classification performance indicators.

[0172] Table 4 Comparison of the classification of the proposed model with other network models

[0173] Network ACC AUC PPV TPR ResNet34 0.6129±0.0385 0.5817±0.0329 0.5580±0.0246 0.5810±0.035 ResNet50 0.5615±0.0640 0.5997±0.0538 0.5780±0.0413 0.6010±0.0550 ResNet101 0.4917±0.0745 0.5500±0.0341 0.5510±0.0384 0.5510±0.0320 Densenet121 0.5378±0.1256 0.5707±0.0512 0.5710±0.0480 0.5710±0.0510 Densenet169 0.5015±0.1452 0.5904±0.0424 0.5880±0.0246 0.5880±0.0430 MCNet 0.6185±0.0713 0.5451±0.0370 0.5373±0.0377 0.5574±0.0366 JMNet 0.7285±0.0256 0.7363±0.0401 0.6790±0.0278 0.7370±0.0400

[0174] Table 5 shows the classification results of ablation experiments on the test set using the proposed network. As can be seen from Table 5, the proposed network improves classification performance by conducting ablation experiments based on a multi-task network. When both the multi-scale convolutional attention module (M-CBAM) and the deep convolutional block (DLCB) are added separately, the classification performance does not improve, but rather decreases. However, when both modules are added simultaneously, classification performance improves. This may be because classification performance is only impacted when the segmentation effect is maximized.

[0175] Table 5 Classification ablation experiments

[0176] M-CBAM DLCB Acc Auc Precision Recall 0.7481 0.7447 0.6850 0.7450 √ 0.7407 0.7481 0.6910 0.7450 √ 0.7111 0.7011 0.6550 0.7000 √ √ 0.7704 0.7840 0.7151 0.7850

[0177] This paper proposes a joint multi-task network (JMNet) for thyroid nodule segmentation and classification in ultrasound images. The segmentation and classification networks are combined through a shared backbone network (FCN). Shallow feature information is shared with the joint network to improve the relevance of shallow features. Furthermore, a multi-scale convolutional attention mechanism (M-CBAM) is proposed, which improves CBAM and is incorporated into the classification network, further enhancing the network's segmentation and classification accuracy.

[0178] In view of the problems that arise in current research, the present invention proposes a method for establishing a thyroid nodule segmentation and classification model based on a multi-task method for combined ultrasound thyroid nodule segmentation and classification. This method uses (Fully Convolutional Networks, FCN) as the backbone shared network, the UNet decoding layer as the segmentation branch network, and ResNet34 as the classification branch network. The backbone shared network uses parameter hard sharing to extract shallow features from the input ultrasound image and shares the extracted features to the two branch networks. In the segmentation branch network, a deep layer convolutional block (DLCB) is first introduced after the backbone shared network to obtain the deep features of the segmentation branch. Secondly, the shallow features obtained in the first feature extraction stage Conv1, the second feature extraction stage Conv2, the third feature extraction stage Conv3, and the fourth feature extraction stage Conv4 are maintained through a jump connection operation with a multiscale convolutional attention module (M-CBAM) to maintain the segmentation edge feature contours of the above four stages and reduce the blurring problem of nodule edges. M-CBAM is combined before and after the classification branch residual module, and the classification performance is optimized through M-CBAM and the residual module.

Claims

1. A method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model, characterized in that: The combined ultrasound thyroid nodule segmentation and classification model uses FCN as the main network framework, and the branch networks are a segmentation branch network based on the UNet decoding layer and a classification branch network based on ResNet34. The method includes: The FCN performs convolution and pooling operations on the image to extract shallow features and obtain the output feature tensor of the FCN; Inputting the output feature tensor of the FCN into the segmentation branch network and the classification branch network respectively; The segmentation branch network is based on the UNet decoding layer, passes the output feature tensor of the FCN through a deep convolution block to extract deep features, then upsamples the deep features, and splices the upsampling result with the feature tensor of each feature extraction stage of the FCN after skip connection to obtain a spliced ​​feature tensor, and finally performs segmentation convolution processing on the spliced ​​feature tensor to obtain a segmentation result, wherein the skip connection is a skip connection based on the multi-scale convolutional attention module M-CBAM; The classification branch network is based on ResNet34, and performs M-CBAM and residual module operations on the output feature tensor of the FCN to obtain the classification result; Training the segmentation branch network and the classification branch network respectively to obtain a loss function of the segmentation branch network and a loss function of the classification branch network; According to the loss function of the segmentation branch network and the loss function of the classification branch network, a total loss function is obtained; according to the total loss function, a preset number of iterations of training are performed to obtain the optimal combined ultrasound thyroid nodule segmentation and classification model to complete the model establishment.

2. The method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model according to claim 1, wherein: The M-CBAM includes a channel attention module and a spatial attention module; The channel attention module first sums the maximum pooling and average pooling through feature splicing, and then trains them through the multi-layer perceptron MLP to enhance the correlation between the two parts of the feature tensor. Then, after the Sigmoid activation function, the output feature tensor is bitwise multiplied with the input feature tensor to obtain the feature tensor output by the channel attention module; The spatial attention module introduces a multi-scale convolution block MSCB to improve the model's ability to process multi-scale nodules. The output feature tensor is then bitwise multiplied with the input feature tensor through a Sigmoid activation function to obtain the feature tensor output by the spatial attention module.

3. The method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model according to claim 2, wherein: The M-CBAM attention mechanism execution steps specifically include: The feature tensor output by each feature extraction stage of the FCN is used as the input feature tensor, and the output feature tensor F1 is obtained through the channel attention module; Passing the output feature tensor F1 through the spatial attention module to obtain an output feature tensor F2; The output feature tensor F2 is multiplied by the feature tensor Fi bit by bit to obtain the feature tensor F3 output by the M-CBAM.

4. The method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model according to claim 3, wherein: The output feature tensor F1 is obtained, specifically including: The output feature tensor F1 is obtained by using the channel attention module expression, where the channel attention module expression is specifically: Among them, F1 is the output feature tensor obtained by the channel attention module, δ is the Sigmoid activation function, MLP is the multi-layer perceptron, MaxPool is the maximum pooling, AvgPool is the average pooling, and F is the input feature tensor. It is bitwise multiplication.

5. The method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model according to claim 3, wherein: The output feature tensor F2 is obtained, specifically including: The output feature tensor F2 is obtained by using the spatial attention module expression, where the spatial attention module expression is specifically: Among them, F2 is the output feature tensor obtained by the spatial attention module, δ is the Sigmoid activation function, MSCB is the multi-scale convolution block, MaxPool is the maximum pooling, AvgPool is the average pooling, and F1 is the output feature tensor obtained by the channel attention module. It is bitwise multiplication.

6. The method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model according to claim 1, characterized in that: The M-CBAM and residual module operations are performed on the output feature tensor of the FCN to obtain the classification result, specifically including: Pass the output feature tensor of the FCN through the M-CBAM to obtain an output feature tensor B1; Pass the output feature tensor B1 through a plurality of the residual modules to obtain an output feature tensor B2; The output feature tensor B2 is passed through the M-CBAM to obtain the output feature tensor B3.

7. The method for establishing a multi-task ultrasound thyroid nodule segmentation and classification model according to claim 6, characterized in that: The residual module includes a first convolutional layer and a second convolutional layer; The first convolutional layer includes 3×3 convolution, batch normalization and sparse activation function; The second convolutional layer includes 3×3 convolution and batch normalization processing; After the output feature tensor B1 passes through the first convolution layer and the second convolution layer in sequence, a first convolution result is obtained; After the output feature tensor B1 passes through 1×1 convolution and BN, a second convolution result is obtained; The first convolution result and the second convolution result are added together, and the added result is passed through a ReLU activation function to obtain the output feature tensor of the residual module.

8. A multi-task ultrasound-based thyroid nodule segmentation and classification method, characterized in that: The thyroid nodule ultrasound image is input into the multi-task ultrasound thyroid nodule segmentation and classification model to obtain the ultrasound thyroid nodule segmentation and classification results.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor runs the computer program stored in the memory, the steps of the method according to any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of computer instructions, and the plurality of computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images