TransUNet-based thyroid nodule ultrasonic image segmentation method and system

By introducing CNN-SETrans encoder and Ghost module in TransUNet, combining the depth-separable convolution and lightweight boundary refinement module, the problem of incomplete feature extraction and difficulty in edge recognition in ultrasonic image segmentation is solved, and high Accurate and stable thyroid nodule segmentation.

CN118941576APending Publication Date: 2024-11-12SUZHOU UNIV
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202410925994.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In ultrasound image segmentation, TransUNet ignores the channel information between images, resulting in incomplete feature extraction and feature loss in jump connections, making it difficult to accurately identify the edge information of thyroid nodules.

Method used

The ultrasonic image segmentation method of thyroid nodules based on TransUNet is introduced, and the CNN-SETrans encoder and Ghost module are introduced. The image information is fully extracted through the SE module and the MSA module in series, and the Ghost module is used to compensate for feature losses during long-distance transmission in the jump connection. , while using depth separable convolution and lightweight boundary refinement modules to improve segmentation accuracy.

Benefits of technology

The feature representation ability of the network is improved, feature loss is compensated, and the accuracy and stability of ultrasonic image segmentation of thyroid nodules is significantly improved, and the calculation cost is not significantly increased.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941576A_ABST
    Figure CN118941576A_ABST
Patent Text Reader

Abstract

The invention discloses a thyroid nodule ultrasound image segmentation method and system based on TransUNet, and relates to the technical field of image segmentation. The method comprises the steps that thyroid nodule ultrasonic images are received and preprocessed, and image data are divided into a training set, a verification set and a test set; and constructing an improved segmentation model based on a TransUNet model network, wherein the improved segmentation model comprises a CNN-SETrans encoder, a decoder and a jump connection. On the basis of a TransUNet network structure, the SE module is introduced into a Transform encoder and is connected with the MSA module in series to construct a SETrans encoder, channel information and spatial information are comprehensively extracted, the feature representation capability of the network can be improved, and by introducing the Ghost module into jump connection, the performance of the network is improved. According to the method, the loss of image features in the long-distance transmission process from the low resolution of the CNN to the high resolution of the decoder can be compensated, the calculation cost cannot be remarkably increased, and meanwhile, the common convolution at the tail of each layer in the decoder is replaced by the DSConv, so that the effectiveness of feature extraction is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a thyroid nodule ultrasound image segmentation method based on TransUNet. Background Art

[0002] Thyroid nodules are tissue masses that appear in the thyroid tissue after abnormal proliferation of thyroid cells. Since thyroid nodules may be a precursor to thyroid cancer, early diagnosis and treatment of thyroid nodules are crucial. In clinical diagnosis, the purpose of thyroid nodule segmentation is to delineate the nodule object of interest from the complex background of the image, which can be used for quantitative diagnosis and morphological analysis of thyroid diseases. However, due to the limitations of imaging, it is difficult for doctors to accurately identify the edge information of the lesion area;

[0003] To solve the above problems, in recent years, scholars in the field of medical image segmentation have mainly conducted research based on U-Net and Transformer. TransUNet has competitiveness in multi-organ segmentation and heart segmentation because it combines the advantages of both. However, in the field of ultrasonic thyroid nodule segmentation, TransUNet ignores the attention to the channel information between images, resulting in incomplete feature extraction and feature loss during long-distance transmission of jump connections. Therefore, we propose a thyroid nodule ultrasonic image segmentation method and system based on TransUNet. Summary of the invention

[0004] The object of the present invention is to provide a thyroid nodule ultrasound image segmentation method and system based on TransUNet, which can maintain high segmentation accuracy and stability.

[0005] To achieve the above object, the present invention provides the following technical solution: a thyroid nodule ultrasound image segmentation method based on TransUNet, comprising the following steps:

[0006] Receive thyroid nodule ultrasound images for preprocessing, and divide the image data into a training set, a validation set, and a test set;

[0007] An improved segmentation model based on the TransUNet model network is constructed. The improved segmentation model includes a CNN-SETrans encoder, a decoder, and a skip connection. The Ghost module is introduced in the skip connection to compensate for the loss of image features during long-distance transmission.

[0008] A lightweight boundary refinement module is introduced at the output of the improved segmentation model to further refine the edges;

[0009] The improved segmentation model is trained using the training set, and the objective function of the improved segmentation model is optimized by minimizing the overall loss function. The parameters of the improved segmentation model are adjusted according to the validation set, and a new round of training is performed until an improved segmentation model that meets the standards is obtained.

[0010] The optimized improved segmentation model is used to identify and segment the ultrasound image to be identified.

[0011] Furthermore, the thyroid nodule ultrasound images are received for preprocessing, and the image data are divided into a training set, a validation set, and a test set, as follows:

[0012] (1) Thyroid nodule ultrasound image data was downloaded based on the TN3K dataset. The image data contained high-quality nodule mask markers from different devices and viewing angles.

[0013] (2) The above image data is divided into a training set, a validation set, and a test set in a ratio of 7:1:2, and data enhancement processing is performed on the training set samples to expand the training samples.

[0014] Furthermore, the CNN-SETrans encoder introduces the SE module based on the Transformer encoder, and connects the SE module with the MSA module in series to construct the SETrans encoder for comprehensive image information extraction. The information extraction is as follows:

[0015] (1) Given an input image x∈R H×W×C , the spatial resolution is H×W, the number of channels is C, the features extracted by CNN are mapped to the image serialization representation, which is reshaped into a patch sequence of dimension (N, D), each patch shape is x patch ∈R P×P×C ;

[0016] (2) Map the features to a new embedding space through patch embedding, and add the position embedding to the patch embedding to obtain the spatial information of each patch. The above process is expressed as:

[0017]

[0018] In the formula, represents linear projection, E position It is a 1D position embedding operation.

[0019] Furthermore, the MSA module is based on the self-attention module, which projects the query vector (Q), key vector (K) and value vector (V) onto D with different learned linear projections h times. q , D k and D vThen, SA is performed in parallel on each projected version of Q, K, and V to produce D v The output value of the dimension, where SA constructs Q, K and V from the input vector. The above process is specifically expressed as:

[0020]

[0021] Q=XW Q

[0022] K=XW K

[0023] V=XW V

[0024] Where X∈R N×D is the input sequence, Q∈R N×D ,K∈R N×D ,V∈R N×D , W Q ∈R D×D , W K ∈R D×D and W V ∈R D×D are all random weights, which are updated as the network training is initialized;

[0025] MSA(Q,K,V)=Concat(h ead1,h ead2,…,h ead h )W O

[0026]

[0027] In the formula, and is the projection parameter matrix.

[0028] Furthermore, the SE module compression operation and excitation operation:

[0029] (1) The compression operation performs global average pooling on the feature map of each channel and compresses the high-dimensional feature representation of each channel into a single value, specifically:

[0030]

[0031] Where x c (i, j) is the pixel value of channel c at position (i, j), and H×W is the size of the feature map;

[0032] (2) The excitation operation uses two consecutive fully connected (FC) layers and an activation function to learn a weight vector, which is used to recalibrate the compressed features. The first FC layer is followed by a ReLU activation function, and the second FC layer is followed by a Sigmoid activation function. The specific expression is as follows:

[0033] s c =σ(g(z c ,W))=σ(W2ReLU(W1z c ))

[0034] Where σ is the Sigmoid function, g represents the activation function, W1 and W2 are the weight matrices of the fully connected layer, and z c is the compressed feature;

[0035] The weight vector s of the excitation output c is used to recalibrate the original feature map x c (i, j), enhance important features and suppress unimportant features by element-by-element multiplication, as shown in the following formula:

[0036] y c (i,j)=s c × c (i,j)

[0037] Where y c (i,j) is the recalibrated feature map.

[0038] Furthermore, the Ghost module uses a parallel mechanism to operate F using 1×1 point convolution to generate a new set of feature maps G; generates more feature maps from G through linear projection, G'=L(G), where L is a linear transformation; concatenates the original feature map F and the newly generated feature map G' to form a wider feature map [F,G']. Finally, these widened feature maps are sent to subsequent network layers for further processing and learning. The above process is specifically expressed as follows:

[0039] G = ReL(BN(Conv1×1))(F)

[0040] G′=ReLU(BN(Conv3×3))(G)

[0041] Output = [F, G′']

[0042] Among them, BN represents batch processing, Output represents the final output, and [,] represents the concatenation operation.

[0043] Furthermore, the decoder comprises three stages, each of which comprises a 3×3 ordinary convolution and a depthwise separable convolution, a transposed convolution layer is used between each stage to enlarge the feature map, the depthwise separable convolution comprises a depthwise convolution and a pointwise convolution, wherein the depthwise convolution independently processes each input channel using a filter; the pointwise convolution integrates the results of the depthwise convolution through a 1×1 convolution operation;

[0044] Depthwise convolution applies a single filter to each input channel, for an input tensor X∈R H×W×C , the depth-wise convolution uses C filters of the same size, and the output also has C channels, which are specifically expressed as follows:

[0045] D c =X c *K c

[0046] Where D c is the output feature map of the cth channel, X c is the input feature map of the cth channel, K c is the cth filter, * indicates convolution operation;

[0047] Point-wise convolution applies 1×1 convolution to combine the output of depth-wise convolution. If there are M output channels, C×M 1×1 filters are used for point-wise convolution. The output P of point-wise convolution can be expressed as:

[0048]

[0049] Where P cm is the output feature map of the mth channel of the cth element, Y c’ is the output feature map of the cth channel of DwConv, K c’c is the PwConv filter, mapping from the c'th output channel to the cth output channel;

[0050] The output of DSConv is thus represented as:

[0051] DSConv = P(D(X))

[0052] Where D(X) represents the depthwise convolution operation on the input X, and P represents the pointwise convolution operation on the output of the depthwise convolution.

[0053] Furthermore, the lightweight boundary refinement module further refines the edge through a combination of two 3×3 ordinary convolutions, an activation function, and a residual connection, as follows:

[0054] Use a 3×3 convolution kernel to convolve to the required number of channels, and then use a 3×3 convolution, ReLU function and residual connection to enhance the edge features to output the segmentation result, as shown below:

[0055] LWBR=Conv3×3+(ReLU(Conv3×3(Conv3×3(X)))).

[0056] Furthermore, the overall loss function is the sum of the BCE Loss loss function and the Dice loss function, as follows:

[0057] L total =L BCE +L dice

[0058] Where L BCE is the BCE Loss function, L dice is the Dice loss function;

[0059] The BCE Loss loss function is as follows:

[0060]

[0061] Where N is the number of samples, L is the loss value, and y i is the true label of the i-th sample, usually taking the value of 0 or 1, y i ' is the predicted value of the i-th sample;

[0062] The dice loss function is as follows:

[0063]

[0064] Where X is the predicted segmentation map, Y is the actual segmentation map, |X∩Y| represents the number of pixels in common with X and Y, and |X| and |Y| represent the total number of pixels in X and Y, respectively.

[0065] According to another aspect of the present invention, the present invention provides a thyroid nodule ultrasound image segmentation system based on TransUNet, comprising:

[0066] A preprocessing module, used for receiving thyroid nodule ultrasound images for preprocessing, and dividing the image data into a training set, a validation set, and a test set;

[0067] A construction module is used to construct an improved segmentation model based on the TransUNet model network. The improved segmentation model includes a CNN-SETrans encoder, a decoder, and a jump connection. The Ghost module is introduced in the jump connection to compensate for the loss of image features during long-distance transmission.

[0068] An edge refinement module is used to introduce a lightweight boundary refinement module at the output end of the improved segmentation model to further refine the edge;

[0069] A training module is used to train the improved segmentation model using the training set, optimize the objective function of the improved segmentation model by minimizing the overall loss function, and adjust the parameters of the improved segmentation model according to the validation set, and conduct a new round of training until an improved segmentation model that meets the standards is obtained;

[0070] The segmentation output module is used to recognize and segment the ultrasound image to be recognized by using the optimized improved segmentation model.

[0071] The present invention has at least the following beneficial effects:

[0072] 1. Based on the TransUNet network structure, the present invention introduces the SE module into the Transformer encoder, and connects it in series with the MSA module to construct the SETrans encoder, which comprehensively extracts channel information and spatial information, can improve the feature representation ability of the network, and by introducing the Ghost module into the jump connection, it can compensate for the loss of image features in the long-distance transmission process from the low resolution of CNN to the high resolution of the decoder, and will not significantly increase the computational cost. At the same time, the deep separable convolution (DSConv) is used to replace the ordinary convolution at the end of each layer in the decoder, which improves the effectiveness of feature extraction;

[0073] 2. The present invention adopts a lightweight boundary refinement module and further refines the edge through a residual connection structure, so as to enhance the accuracy and stability of the segmentation results.

[0074] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a schematic diagram of the overall structure of the improved segmentation model in the present invention;

[0076] Figure 2 It is a structural schematic diagram of the SETrans encoder in the present invention;

[0077] Figure 3 It is a schematic diagram of the structure of the Ghost module in the present invention;

[0078] Figure 4 Schematic diagram of the structure of the depthwise separable convolution in the present invention;

[0079] Figure 5 It is a structural schematic diagram of the lightweight boundary refinement module in the present invention;

[0080] Figure 6It is a schematic diagram for qualitative analysis and comparison of ablation experiment results of the method described in the present invention. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0082] Embodiment 1:

[0083] See also Figure 1-6 The present invention provides a technical solution: a thyroid nodule ultrasound image segmentation method based on TransUNet, comprising the following steps:

[0084] S1. Receive thyroid nodule ultrasound images for preprocessing and divide the image data into training set, validation set and test set, as follows:

[0085] (1) Thyroid nodule ultrasound image data was downloaded based on the TN3K dataset. The image data contained high-quality nodule mask markers from different devices and viewing angles.

[0086] (2) Divide the above image data into a training set, a validation set, and a test set in a ratio of 7:1:2, perform data augmentation on the training set samples, and expand the training samples;

[0087] Furthermore, data enhancement processing includes flipping, random rotation and scaling to alleviate the overfitting phenomenon of deep neural networks.

[0088] It should be noted that the TN3K dataset is a challenging dataset provided by Southern Medical University specifically for thyroid nodule segmentation in ultrasound images. The dataset contains 3493 ultrasound images with pixel-level annotations.

[0089] S2. Construct an improved segmentation model based on the TransUNet model network. The improved segmentation model includes a CNN-SETrans encoder, a decoder, and a jump connection. The Ghost module is introduced in the jump connection to compensate for the loss of image features during long-distance transmission. The details are as follows:

[0090] According to the technical solution of this embodiment, in order to solve the problem that the Transformer encoder ignores the extraction of channel information, the encoder of the improved segmentation model introduces the SE module in the Transformer encoder, and connects the SE module with the MSA (multi-head self-attention module) module in series to construct the SETrans encoder for comprehensive extraction of image information, as follows:

[0091] Given an input image x∈R H×W×C , the spatial resolution is H×W, and the number of channels is C. The feature map extracted by CNN is converted into an image serialization representation, which is reshaped into a patch sequence of dimension (N, D), each patch shape is x patch ∈R P×P×C , Subsequently, the features are mapped to the new embedding space through patch embedding, and the position embedding is added to the patch embedding to obtain the spatial information of each patch. The above process can be expressed as:

[0092]

[0093] In the formula, represents linear projection, E position It is a 1D position embedding operation;

[0094] According to the technical solution of this embodiment, the multi-head self-attention module (MSA) is based on the self-attention module (SA) and projects the query vector (Q), the key vector (K) and the value vector (V) onto D using different learning linear projections h times. q , D k and D v Then, SA is performed in parallel on each projected version of Q, K, and V to produce D v The output value of the dimension, where SA constructs Q, K and V from the input vector, the above process can be expressed as:

[0095]

[0096] Q=XW Q

[0097] K=XW K

[0098] V=XW V

[0099] Where X∈R N×D is the input sequence, Q∈R N×D ,K∈R N×D ,V∈R N×D , W Q ∈R D×D , W K ∈RD×D and W V ∈R D×D are all random weights, which are updated as the network training is initialized;

[0100] MSA(Q,K,V)=Concat(h ead1,h ead2,…,h ead h )W O

[0101]

[0102] In the formula, and is the parameter matrix of the projection;

[0103] According to the technical solution of this embodiment, the SE module is conducive to improving network performance. It enhances the feature representation ability of the network by explicitly modeling the relationship between channels. The SE module is relatively lightweight and does not significantly increase the computational burden of the network. The SE module consists of two main operations: compression and excitation.

[0104] (1) The compression operation performs global average pooling on the feature map of each channel, compressing the high-dimensional feature representation of each channel into a single value. This process can be expressed as follows:

[0105]

[0106] Where x c (i, j) is the pixel value of channel c at position (i, j), and H×W is the size of the feature map;

[0107] (2) The excitation operation uses two consecutive fully connected (FC) layers and an activation function to learn a weight vector, which is used to recalibrate the compressed features. The first FC layer is followed by a ReLU activation function, and the second FC layer is followed by a Sigmoid activation function, which is expressed as follows:

[0108] s c =σ(g(z c ,W))=σ(W2ReLU(W1z c ))

[0109] Where σ is the Sigmoid function, g represents the activation function, W1 and W2 are the weight matrices of the fully connected layer, and z c is the compressed feature;

[0110] The weight vector s of the excitation output c is used to recalibrate the original feature map x c (i, j), enhance important features and suppress unimportant features by element-by-element multiplication, as shown in the following formula:

[0111] y c (i,j)=s c × c (i,j)

[0112] Where y c (i,j) is the recalibrated feature map;

[0113] For the technical solution of this embodiment, the Ghost module is used to improve the capacity and performance of the model by increasing the width of the network without significantly increasing the computational cost. The Ghost module uses a parallel mechanism to operate F using a 1×1 point convolution to generate a new set of feature maps G; more feature maps are generated from G through linear projection, G'=L(G), where L is a linear transformation; the original feature map F and the newly generated feature map G' are spliced ​​to form a wider feature map [F, G'], and finally these widened feature maps are sent to subsequent network layers for further processing and learning. The above process can be expressed as:

[0114] G = ReLU(BN(Conv1×1))(F)

[0115] G′=ReLU(BN(COnv3×3))(G)

[0116] Output=[F,G′]

[0117] In the formula, BN represents batch processing, Output represents the final output, and [,] represents the concatenation operation.

[0118] According to the technical solution of this embodiment, the decoder uses deep separable convolution (DSConv) to reconstruct the high-resolution feature map, which includes three stages, each of which consists of a 3×3 ordinary convolution and a DSConv. A transposed convolution layer is used between each stage to increase the feature map and gradually restore the resolution of the image.

[0119] Depthwise separable convolution (DSConv) consists of 1 depthwise convolution (DwConv) and 1 pointwise convolution (PwConv), one layer for filtering and the other for combining;

[0120] Furthermore, DwConv processes each input channel independently using a filter, and then PwConv integrates the results of depth-wise convolution through a 1×1 convolution operation, which helps to significantly reduce the effect of computation and model size while ensuring the network's ability to learn features;

[0121] DwConv applies a single filter to each input channel, for an input tensor X∈R H×W×C, DwConv uses C filters of the same size, and the output also has C channels, which can be expressed as follows:

[0122] D c =X c *K c

[0123] Where D c is the output feature map of the cth channel, X c is the input feature map of the cth channel, K c is the cth filter, * indicates convolution operation;

[0124] Furthermore, PwConv applies 1×1 convolution to combine the output of DwConv, with the aim of expanding the output channels of DwConv to the desired output channels while keeping the spatial dimension unchanged. If there are M output channels, C×M 1×1 filters are used for PwConv. The output P of PwConv can be expressed as:

[0125]

[0126] Where P cm is the output feature map of the mth channel of the cth element, Y c’ is the output feature map of the cth channel of DwConv, K c’c is the PwConv filter, mapping from the c'th output channel to the cth output channel. In summary, the output of DSConv can be expressed as:

[0127] DSConv = P(D(X))

[0128] Where D(X) represents the depth convolution operation on the input X, and P represents the point-by-point convolution operation on the output of the depth convolution;

[0129] S3. A lightweight boundary refinement module is introduced at the output end of the improved segmentation model to further refine the edge;

[0130] According to the technical solution of this embodiment, the lightweight boundary refinement module further refines the edge through two 3×3 ordinary convolutions, a combination of an activation function and a residual connection, as follows:

[0131] Use a 3×3 convolution kernel to convolve to the required number of channels, and then use a 3×3 convolution, ReLU function and residual connection to enhance the edge features to output the segmentation result, as shown below:

[0132] LWBR=Conv3×3+(ReLU(Conv3×3(Conv3×3(X))));

[0133] S4. Use the training set to train the improved segmentation model, minimize the overall loss function as the optimization objective function of the improved segmentation model, and adjust the parameters of the improved segmentation model according to the validation set, and perform a new round of training until an improved segmentation model that meets the standards is obtained;

[0134] For the technical solution of this embodiment, the overall loss function is the sum of the BCE Loss function and the Dice loss function, which is as follows:

[0135] L total =L BCE +L dice

[0136] Where L BCE is the BCE Loss function, L dice is the Dice loss function;

[0137] The BCE Loss loss function is as follows:

[0138]

[0139] Where N is the number of samples, L is the loss value, and y i is the true label of the i-th sample, usually taking the value of 0 or 1, y i ' is the predicted value of the i-th sample;

[0140] The dice loss function is as follows:

[0141]

[0142] Where X is the predicted segmentation map, Y is the actual segmentation map, |X∩Y| represents the number of pixels in common with X and Y, and |X| and |Y| represent the total number of pixels in X and Y respectively;

[0143] It should be noted that the improved segmentation model of this embodiment is based on the PyTorch 2.1.0 deep learning framework, the GPU configuration is NVIDIA GeForce RTX 4090 with 32GB memory, a total of 150 cycles are trained, a multivariate learning rate strategy is used, the learning rate lr = 0.01, the batch_size value is set to 6, and the SGD optimization algorithm is specifically used. The input image size is 512×512;

[0144] S5. Use the optimized improved segmentation model to identify and segment the ultrasound image to be identified.

[0145] In order to quantitatively evaluate the performance of the method proposed in this embodiment, two commonly used evaluation indicators in medical image segmentation, Dice coefficient (DSC) and intersection over union (IoU), are used as the measurement criteria for the experimental results. DSC is a function that evaluates the similarity between the predicted results and the manual annotations, and is defined as:

[0146]

[0147] Where X is the predicted segmentation map, Y is the actual segmentation map, |X∩Y| represents the number of pixels in common with X and Y, |X| and |Y| represent the total number of pixels in X and Y respectively. This embodiment retains the best training weight based on the Dice coefficient to facilitate subsequent prediction of the test set data. IoU is the ratio of the overlap between the predicted segmentation area and the actual annotated area to the joint area of ​​the two areas, which is defined as:

[0148]

[0149] Where PS is the predicted area, GT is the true area, |GT∩PS| represents the intersection area of ​​the predicted segmentation area and the true segmentation area (that is, the number of overlapping pixels), and |GT∪PS| represents the union area of ​​the predicted segmentation area and the true segmentation area (that is, the total number of pixels, including all pixels in the predicted and true areas).

[0150] In order to prove the effectiveness of the module, the SGDLTU-Net proposed by us is selected as the basic network, and the quantitative analysis of the corresponding ablation experiment results is shown in Table 1. The results show that the SE, Ghost module and DSConv proposed in this embodiment are all effective. As shown in Table 1, as more components are removed from the network structure, the means of DSC and IoU generally show a downward trend, which shows that these components help improve the performance of the model; SGDLTU-Net (TransUNet+SETrans+DSConv+Ghost+LWBR) shows the highest means (0.8271 and 0.7341) in both DSC and IoU, which means that it performs best among all variants. When the SETrans component is removed, the IoU result of the model drops severely, indicating that the SETrans module proposed in this paper is very effective in improving the segmentation quality. The addition of DSConv has a positive impact on the performance, because the performance is improved from TransUNet to TransUNet+SETrans+DSConv. The addition of the Ghost module brings performance improvement on DSC, and the addition of LWBR brings significant performance improvement on DSC. The qualitative analysis of the ablation experiment results is shown in Table 1. Figure 6 As shown, it further indicates that each module of this embodiment is effective.

[0151] In this embodiment, the proposed method is compared with U-Net, UNeXt, TransUNet and TRFENet. The segmentation results of different methods are shown in Table 2. The results show that the method in this embodiment has the highest Dice coefficient and IoU coefficient for all thyroid nodule segmentations.

[0152] Table 1 Comparison of segmentation results of ablation experiments (all models are pre-trained)

[0153]

[0154]

[0155] Table 2 Performance comparison with other existing medical image segmentation models (all models are not pre-trained)

[0156] Model DSC↑ IoU↑ U-Net 0.6408±0.2197 0.6587±0.2043 UNeX 0.7691±0.2695 0.6296±0.2525 TransUNet 0.7669±0.2215 0.6636±0.2356 TRFENet 0.6940±0.2444 0.5770±0.2482 SGDLTU-Net(ours) 0.8114±0.2006 0.7194±0.2308

[0157] To summarize, the technical solution of this embodiment has a jump-connected encoder-decoder architecture TransUNet, which fuses the channel attention module with the Transformer for comprehensive encoding, places the Ghost module in the jump connection to compensate for the loss of image features during the long-distance transmission from the low resolution of CNN to the high resolution of the decoder, and replaces the ordinary convolution at the end of each layer of the decoder with a depthwise separable convolution to reduce the number of parameters. At the same time, a lightweight boundary refinement module is placed at the output end of the model to improve the accuracy of thyroid nodule segmentation. Compared with the traditional segmentation network, the method proposed in this embodiment has higher accuracy and stability in the automatic segmentation of thyroid nodule ultrasound images.

[0158] Embodiment 2:

[0159] The present invention provides a thyroid nodule ultrasound image segmentation system based on TransUNet, comprising:

[0160] A preprocessing module, used for receiving thyroid nodule ultrasound images for preprocessing, and dividing the image data into a training set, a validation set, and a test set;

[0161] A construction module is used to construct an improved segmentation model based on the TransUNet model network. The improved segmentation model includes a CNN-SETrans encoder, a decoder, and a jump connection. The Ghost module is introduced in the jump connection to compensate for the loss of image features during long-distance transmission.

[0162] An edge refinement module is used to introduce a lightweight boundary refinement module at the output end of the improved segmentation model to further refine the edge;

[0163] A training module is used to train the improved segmentation model using the training set, optimize the objective function of the improved segmentation model by minimizing the overall loss function, and adjust the parameters of the improved segmentation model according to the validation set, and conduct a new round of training until an improved segmentation model that meets the standards is obtained;

[0164] The segmentation output module is used to recognize and segment the ultrasound image to be recognized by using the optimized improved segmentation model.

[0165] Specifically, the above-mentioned preprocessing module, construction module, edge refinement module, training module and segmentation output module can be embedded in a computer processing system. The computer calls the above-mentioned modules to complete the task of equivalent modeling according to the above-mentioned TransUNet-based thyroid nodule ultrasound image segmentation method; the above-mentioned preprocessing module, construction module, edge refinement module, training module and segmentation output module can perform operations according to the specific steps given in the above-mentioned TransUNet-based thyroid nodule ultrasound image segmentation method.

[0166] It should be noted that it should be understood that the division of the various modules of the above system is only the division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated, and these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the preprocessing module can be a separately established processing element, or it can be integrated in a certain chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a certain processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by the hardware integrated logic circuit in the processor element or the instructions in the form of software.

[0167] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0168] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0169] For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. When an element is referred to as being "assembled on", "installed on", "fixed on" or "set on" another element, it can be directly on the other element or there can also be a centered element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be a centered element at the same time. The terms "vertical", "horizontal", "up", "down", "left", "right" and similar expressions used herein are only for illustrative purposes and are not intended to be the only implementation method.

[0170] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0171] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

Claims

1. A thyroid nodule ultrasound image segmentation method based on TransUNet, characterized in that: The following steps are involved: Receive thyroid nodule ultrasound images for preprocessing, and divide the image data into a training set, a validation set, and a test set; An improved segmentation model based on the TransUNet model network is constructed. The improved segmentation model includes a CNN-SETrans encoder, a decoder, and a skip connection. The Ghost module is introduced in the skip connection to compensate for the loss of image features during long-distance transmission. A lightweight boundary refinement module is introduced at the output of the improved segmentation model to further refine the edges; The improved segmentation model is trained using the training set, and the objective function of the improved segmentation model is optimized by minimizing the overall loss function. The parameters of the improved segmentation model are adjusted according to the validation set, and a new round of training is performed until an improved segmentation model that meets the standards is obtained. The optimized improved segmentation model is used to identify and segment the ultrasound image to be identified.

2. The thyroid nodule ultrasound image segmentation method based on TransUNet according to claim 1, characterized in that: Receive thyroid nodule ultrasound images for preprocessing, and divide the image data into training set, validation set, and test set, as follows: (1) Thyroid nodule ultrasound image data was downloaded based on the TN3K dataset. The image data contained high-quality nodule mask markers from different devices and viewing angles. (2) The above image data is divided into a training set, a validation set, and a test set in a ratio of 7:1:2, and data enhancement processing is performed on the training set samples to expand the training samples.

3. The thyroid nodule ultrasound image segmentation method based on TransUNet according to claim 2, characterized in that: The CNN-SETrans encoder introduces the SE module based on the Transformer encoder, and connects the SE module with the MSA module in series to construct the SETrans encoder, which is used to fully extract image information. The information extraction is as follows: (1) Given an input image x∈R H×W×C , the spatial resolution is H×W, the number of channels is C, the features extracted by CNN are mapped to the image serialization representation, which is reshaped into a patch sequence of dimension (N, D), each patch shape is x patch ∈R P×P×C ; (2) Map the features to a new embedding space through patch embedding, and add the position embedding to the patch embedding to obtain the spatial information of each patch. The above process is expressed as: In the formula, i=1,2,...,N, represents linear projection, E position It is a 1D position embedding operation.

4. The method for segmenting thyroid nodules using ultrasound images based on TransUNet according to claim 3, characterized in that: The MSA module is based on the self-attention module, which projects the query vector (Q), key vector (K) and value vector (V) onto D with different learned linear projections h times. q , D k and D v Then, SA is performed in parallel on each projected version of Q, K, and V to produce D v The output value of the dimension, where SA constructs Q, K and V from the input vector. The above process is specifically expressed as: Q=XW Q K=XW K V=XW V Where X∈R N×D is the input sequence, Q∈R N×D , K∈R N×D , V∈R N×D , W Q ∈R D×D , W K ∈R D×D and W V ∈R D×D are all random weights, which are updated as the network training is initialized; MSA(Q,K,V)=Concat(head1,head2,…,head h )w O In the formula, and is the projection parameter matrix.

5. The thyroid nodule ultrasound image segmentation method based on TransUNet according to claim 4, characterized in that: The SE module compression operation and excitation operation: (1) The compression operation performs global average pooling on the feature map of each channel and compresses the high-dimensional feature representation of each channel into a single value, specifically: Where x c (i, j) is the pixel value of channel c at position (i, j), and H×W is the size of the feature map; (2) The excitation operation uses two consecutive fully connected (FC) layers and an activation function to learn a weight vector, which is used to recalibrate the compressed features. The first FC layer is followed by a ReLU activation function, and the second FC layer is followed by a Sigmoid activation function. The specific expression is as follows: s c =σ(g(z c ,W))=σ(W2ReLU(W1Z c )) Where σ is the Sigmoid function, g represents the activation function, W1 and W2 are the weight matrices of the fully connected layer, and z c is the compressed feature; The weight vector s of the excitation output c is used to recalibrate the original feature map x c (i, j), enhance important features and suppress unimportant features by element-by-element multiplication, as shown in the following formula: y c (i,j)=s c ×x c (i,j) Where y c (i, j) is the recalibrated feature map.

6. The method for segmenting thyroid nodules ultrasound images based on TransUNet according to claim 4, characterized in that: The Ghost module operates F using a 1×1 point convolution through a parallel mechanism to generate a set of new feature maps G; Generate more feature maps from G through linear projection, G'=L(G), where L is a linear transformation; concatenate the original feature map F and the newly generated feature map G' to form a wider feature map [F, G']. Finally, these widened feature maps are sent to the subsequent network layers for further processing and learning. The above process is specifically expressed as follows: G = ReLU(BN(Conv1×1))(F) G′=ReLU(BN(Conv3×3))(G) Output = [F, G′] Among them, BN represents batch processing, Output represents the final output, and [,] represents the concatenation operation.

7. The method for segmenting thyroid nodules ultrasound images based on TransUNet according to claim 6, characterized in that: The decoder consists of three stages, each of which includes a 3×3 normal convolution and a depthwise separable convolution. A transposed convolution layer is used between each stage to enlarge the feature map. The depthwise separable convolution consists of a depthwise convolution and a pointwise convolution. The depthwise convolution independently processes each input channel using a filter; the pointwise convolution integrates the results of the depthwise convolution through a 1×1 convolution operation. Depthwise convolution applies a single filter to each input channel, for an input tensor X∈R H×W×C , the depth-wise convolution uses C filters of the same size, and the output also has C channels, which are specifically expressed as follows: D c =X c *K c Where D c is the output feature map of the cth channel, X c is the input feature map of the cth channel, K c is the cth filter, * indicates convolution operation; Point-wise convolution applies 1×1 convolution to combine the output of depth-wise convolution. If there are M output channels, C×M 1×1 filters are used for point-wise convolution. The output P of point-wise convolution can be expressed as: Where P cm is the output feature map of the mth channel of the cth element, Y c’ is the output feature map of the cth channel of DwConv, K c’c is the PwConv filter, mapping from the c'th output channel to the cth output channel; The output of DSConv is thus represented as: DSConv = P(D(X)) Where D(X) represents the depthwise convolution operation on the input X, and P represents the pointwise convolution operation on the output of the depthwise convolution.

8. The method for segmenting thyroid nodules ultrasound images based on TransUNet according to claim 7, characterized in that: The lightweight boundary refinement module further refines the edge through two 3×3 ordinary convolutions, a combination of an activation function and a residual connection, as follows: Use a 3×3 convolution kernel to convolve to the required number of channels, and then use a 3×3 convolution, ReLU function and residual connection to enhance the edge features to output the segmentation result, as shown below: LWBR=Conv3×3+(ReLU(conv3×3(conv3×3(X)))).

9. The method for segmenting thyroid nodules ultrasound images based on TransUNet according to claim 8, characterized in that: The overall loss function is the sum of the BCE Loss loss function and the Dice loss function, as follows: L total =L BCE +L dice Where L BCE is the BCE Loss function, L dice is the Dice loss function; The BCE Loss loss function is as follows: Where N is the number of samples, L is the loss value, and y i is the true label of the i-th sample, usually taking the value of 0 or 1, y i ' is the predicted value of the i-th sample; The dice loss function is as follows: Where X is the predicted segmentation map, Y is the actual segmentation map, |X∩Y| represents the number of pixels in common with X and Y, and |X| and |Y| represent the total number of pixels in X and Y, respectively.

10. A thyroid nodule ultrasound image segmentation system based on TransUNet, characterized in that: include: A preprocessing module, used for receiving thyroid nodule ultrasound images for preprocessing, and dividing the image data into a training set, a validation set, and a test set; A construction module is used to construct an improved segmentation model based on the TransUNet model network. The improved segmentation model includes a CNN-SETrans encoder, a decoder, and a jump connection. The Ghost module is introduced in the jump connection to compensate for the loss of image features during long-distance transmission. An edge refinement module is used to introduce a lightweight boundary refinement module at the output end of the improved segmentation model to further refine the edge; A training module is used to train the improved segmentation model using the training set, optimize the objective function of the improved segmentation model by minimizing the overall loss function, and adjust the parameters of the improved segmentation model according to the validation set, and conduct a new round of training until an improved segmentation model that meets the standards is obtained; The segmentation output module is used to recognize and segment the ultrasound image to be recognized by using the optimized improved segmentation model.

Citation Information

Cited By

  • Cervical cancer clinical target region segmentation method guided by anatomical prior position information

    CN119810118A

  • Thyroid nodule segmentation method based on prior attention and auxiliary guidance

    CN119919655A

  • Analog circuit early fault diagnosis method based on subsequence division and Transform

    CN120951916A

  • Image segmentation method, system and device

    CN121788837A