Thyroid ultrasound image segmentation method based on high-resolution neural network

Abstract

The invention discloses a thyroid ultrasound image segmentation method based on a high-resolution neural network. The method comprises the following steps: extracting data features by using a trunk neural network model staying at a high resolution rate; a plurality of open thyroid ultrasound data sets of different channels are adopted for training; carrying out modeling on global information by using a converter module on a bottom abstract feature layer; and fusing the feature maps of different scales, sending the fused feature maps into a pyramid module, extracting information in a larger range by using hole convolution, and finally outputting the fused feature maps to a classifier to classify each pixel point in an original image so as to complete image segmentation. According to the thyroid ultrasound image segmentation method, the problem of high image quality requirement on the thyroid ultrasound image segmentation task in the prior art is solved by taking staying at high resolution as a core idea, and meanwhile, the segmentation accuracy is remarkably improved on the premise of increasing less calculation amount and performance overhead.

Description

Technical Field

[0001] This invention relates to a technology in the field of deep learning, specifically a method for segmenting thyroid ultrasound images based on a high-resolution neural network. Background Technology

[0002] In thyroid ultrasound image segmentation, the segmentation results are typically strongly correlated with the ultrasound image acquisition equipment and the operator's skill. Traditional image processing methods utilize the principle that the boundary pixels between the target region and its surrounding regions exhibit abrupt changes, employing local differential operators to determine whether a point is an edge node. However, due to variations in the quality of the image acquisition equipment and the skill level of the operators, these methods can lead to misclassification and difficulty in analyzing regional textures. Consequently, when data quality is poor, the segmentation results are significantly less than expected. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes a thyroid ultrasound image segmentation method based on a high-resolution neural network. By focusing on high resolution as its core principle, this method tackles the challenge of high image quality requirements in current thyroid ultrasound image segmentation tasks, while significantly improving segmentation accuracy with minimal computational and performance overhead.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a thyroid ultrasound image segmentation method based on a high-resolution neural network. The method extracts data features using a high-resolution backbone neural network model; trains the image using multiple publicly available thyroid ultrasound datasets from different sources; models global information using a transformer module at the bottom abstract feature layer; fuses feature maps of different scales and feeds them into a pyramid module using dilated convolution to extract information over a wider range; finally, the output is sent to a classifier to classify each pixel in the original image, thus completing the image segmentation.

[0006] The high-resolution neural network model includes: a high-resolution feature extraction backbone network, a global modeling transformer module, a multi-scale fusion module, and a pyramid prediction module. The feature extraction backbone network consists of four stages. Each stage, while maintaining the existing resolution branch, obtains an additional branch with a lower resolution by controlling the convolution stride, thus obtaining feature maps of the image at different scales. The global modeling transformer module calculates the self-attention of the low-level abstract feature maps to obtain the relationship between each pixel and any pixel in the image, enabling the model to model pixels globally. The multi-scale fusion module uses upsampling and downsampling to fuse features at different scales at the end of stages representing different depths, better utilizing information from features at different scales. The pyramid prediction module, adhering to the overall invention's concept, concatenates feature maps at different scales and calculates the probability of each pixel using dilated convolutions at different scales, thereby completing the final segmentation task.

[0007] The high-resolution feature extraction backbone network includes a feature extraction unit and a downsampling unit. The feature extraction unit combines four convolutional modules in a serial manner to extract feature information. The downsampling unit reduces the resolution by using convolutional modules with controlled strides. The backbone network combines the feature extraction unit and the downsampling unit into four stages. The first stage only has a feature extraction unit at the original resolution. The second stage uses the downsampling unit to add an additional 1 / 2 resolution feature extraction unit. Similarly, the third stage adds an additional 1 / 4 resolution feature extraction unit. The fourth stage only adds a branch starting with a 1 / 8 resolution feature map to connect to other modules.

[0008] The global modeling transformer module further includes a global modeling transformer weighted by x and y axis dimensional information, which establishes a global connection for the abstract feature map at 1 / 8 resolution.

[0009] The multi-scale fusion module includes an upsampling unit and a convolutional unit. The upsampling unit scales the feature maps with different resolutions generated after each stage in the backbone network to the same resolution, and then fuses the features at each scale through the convolutional unit.

[0010] The pyramid prediction module includes: a dilated convolution unit, a convolution unit, and a classifier unit. The dilated convolution unit performs dilated convolution processing of different sizes based on the four feature maps of different resolutions generated in the fourth stage to obtain a wider range of information connections. The convolution unit fuses the final result of the dilated convolution and outputs it to the classifier unit for binary classification to obtain a pixel-dimensional prediction map. Technical effect

[0011] This invention applies a high-resolution neural network model to thyroid ultrasound image segmentation. To improve the performance of the high-resolution neural network model under poor image quality conditions, a transformer is introduced in the low-level feature layer to replace traditional convolutional feature extraction. Compared with existing technologies, this invention significantly improves the accuracy of thyroid ultrasound image segmentation, making it feasible to apply artificial intelligence-assisted clinical diagnosis in the field of thyroid ultrasound treatment. This method has strong generalization ability across different ultrasound devices, and computer-aided diagnosis no longer requires high image quality. Since this model method is implemented based on the mmsegmentation framework, multi-GPU distributed training can be easily implemented in practical engineering applications, thereby improving the ability and efficiency of processing large-scale data. Attached Figure Description

[0012] Figure 1 This is a flowchart of the present invention;

[0013] Figure 2 is The main training flowchart of this invention;

[0014] Figure 3 This is a schematic diagram of the converter module of the present invention;

[0015] Figure 4 This is a schematic diagram illustrating the prediction effect of the present invention. Detailed Implementation

[0016] like Figure 1 As shown, this embodiment illustrates a segmentation method based on high-resolution thyroid ultrasound images, comprising two stages: offline training of the model and online prediction.

[0017] Step 1, Data Acquisition: Data acquisition during the offline training phase can utilize publicly available datasets or internal datasets owned by hospitals, companies, or organizations. For the online prediction phase, data acquisition involves clinicians using ultrasound equipment to examine patients. The resulting medical image data is stored in the hospital's data center, de-identified, exported, and then fed into subsequent processes.

[0018] Step 2, Data Labeling: This requires professional and experienced clinicians to diagnose the ultrasound images and use computer-aided software to outline the nodule region. The resulting target region mask is then processed to obtain labeled data that can be used for training.

[0019] Step 3, Data Preprocessing: First, the pixel distribution of each image is statistically analyzed, and then histogram normalization is performed to improve image contrast and enhance the distinction between target areas and background pixels. Then, irrelevant pixels at image edges, such as some parameters inherent in ultrasound equipment, are cropped to avoid unnecessary calculations and interference with training.

[0020] Step 4, Data Augmentation: By performing simple operations such as flipping, rotating, cropping, scaling, and translating on the original data images, the following functions can be achieved: avoiding overfitting and making it more difficult for the model to learn features unrelated to the target; improving the robustness of the model and reducing the model's requirements for image quality; increasing training data; and avoiding imbalanced sample distribution.

[0021] Step 5, Data Training: Input the data-enhanced images in batches.

[0022] Step 6, Data Prediction: The data collected in the production environment is pulled from the data center to the server, preprocessed, and then the optimal model parameters obtained in the training phase are used for prediction. The prediction results are then displayed as a mask over the original image.

[0023] Figure 2 shows the structure of this neural network model, which includes: a high-resolution backbone feature extraction module, a global modeling transformer module, a multi-scale fusion module, and a pyramid prediction head module.

[0024] The high-resolution backbone feature extraction module maintains high-resolution representation during training, ensuring no loss of pixel accuracy. It starts with a high-resolution subnet as the first stage, progressively adding more subnets from high to low resolution through downsampling, forming more stages, and then connecting these multi-resolution subnets. There are four stages in total, each with one more subnet branch of half the resolution than the previous stage. By connecting the high-resolution to low-resolution subnets in parallel, it avoids the information loss associated with traditional low-resolution to high-resolution reconstruction, resulting in a more spatially accurate predicted image.

[0025] Global Modeling Transformer Module: This module performs global modeling of deep feature information during training. In the context of thyroid ultrasound image segmentation, the image displays the human thyroid gland and surrounding tissues. Different tissue regions are spatially related. For each feature map input to this module, each pixel represents 1 / 64 of the feature information of the original image, which is an appropriate granularity to represent different tissue regions in the original image. By inputting the regional feature information map into this module, a global model is established for the data to extract the relationships between different tissue regions, thereby better identifying and segmenting the target region.

[0026] Multi-scale fusion module: Traditional scale fusion modules restore the size of the target feature map from a single feature map by upsampling and downsampling, and then fuse them by stitching. This module, however, samples and fuses feature maps of different resolutions with the same depth and high similarity generated at each stage through convolution. It uses the low-resolution feature map to enhance the representation of the high-resolution feature map, and conversely, it uses the completeness of the high-resolution feature map to enhance the detailed information of the low-resolution feature map.

[0027] Pyramid Prediction Module: This module receives four feature maps of different resolutions, uses dilated convolution to further extract information over a larger range, then inputs them into a softmax classifier, outputs a prediction result for each pixel, and finally calculates the loss function and uses gradient backpropagation for training.

[0028] like Figure 3 The diagram shows the transformer module in this neural network model. Unlike traditional transformer modules, this module combines the advantages of transformer modules and convolutional layers. By assembling tensors, it provides a deep understanding of the local and global abstract features contained in different regions of thyroid ultrasound images. This effectively integrates local details and overall structural information in thyroid ultrasound images, contributing to improved segmentation accuracy and robustness.

[0029] Through specific practical experiments: the mmsegmentation framework was used as the framework for model implementation; the cross-entropy loss function was set; after calculating the distribution of the red, green, and blue channel values ​​of the original data samples, histogram equalization was performed on the original data; irrelevant rows and columns were cropped during preprocessing; data augmentation operations such as flipping, selection, and scaling were performed during training; the stochastic gradient descent algorithm with a variable learning rate was selected as the optimization function; and the random number seed was 9215888155. After preparing the experiment according to the above steps, it was trained on the NVIDIA 3090 image processor, the converged model was saved, and predictions were made on the pre-prepared test set data to obtain the prediction results.

[0030] like Figure 4 The diagram shows the prediction results of thyroid ultrasound images using the method of this invention. It illustrates four thyroid ultrasound images and their predicted images, with nodules in the images exhibiting four different characteristics: large, medium, small, and irregular. As can be seen from the diagram, this method achieves good results in segmenting thyroid ultrasound images with various characteristics.

[0031] Compared to existing technologies, the performance improvements of this device / method (i.e., the effects achieved by employing new technologies in the aforementioned stages) are as follows: By keeping the backbone feature extraction network at high resolution, complete pixel information is preserved. This results in higher prediction accuracy for subsequent predictions compared to traditional convolutional neural networks that lose pixel information during feature extraction. Consequently, this method performs well even with low-quality images. Furthermore, by using a transformer to perform long-distance modeling on the low-level abstract features, this method provides a more comprehensive understanding of the spatial topology of various tissues in the image compared to traditional convolutional neural networks that only possess local receptive fields, thus achieving more accurate segmentation capabilities.

[0032] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A method for thyroid ultrasound image segmentation based on high-resolution neural networks, characterized in that, Data features are extracted using a high-resolution backbone neural network model; multiple publicly available thyroid ultrasound datasets from different sources are used for training; a transformer module is used to model global information in the bottom abstract feature layer; feature maps of different scales are fused and fed into a pyramid module to extract information over a wider range using dilated convolution; finally, the output is sent to a classifier to classify each pixel in the original image, thus completing image segmentation.

2. The thyroid ultrasound image segmentation method based on high-resolution neural networks according to claim 1, characterized in that, The high-resolution neural network model includes: a high-resolution feature extraction backbone network, a global modeling transformer module, a multi-scale fusion module, and a pyramid prediction module. The feature extraction backbone network consists of four stages. Each stage, while maintaining the existing resolution branch, obtains an additional branch with a lower resolution by controlling the convolution stride, thus obtaining feature maps of the image at different scales. The global modeling transformer module calculates the self-attention of the low-level abstract feature maps to obtain the relationship between each pixel and any pixel in the image, enabling the model to model pixels globally. The multi-scale fusion module uses upsampling and downsampling to fuse features at different scales at the end of stages representing different depths, better utilizing information from features at different scales. The pyramid prediction module, adhering to the overall invention's concept, concatenates feature maps at different scales and calculates the probability of each pixel using dilated convolutions at different scales, thereby completing the final segmentation task.

3. The thyroid ultrasound image segmentation method based on high-resolution neural networks according to claim 2, characterized in that, The feature extraction backbone network includes a feature extraction unit and a downsampling unit. The feature extraction unit combines four convolutional modules in a serial manner to extract feature information. The downsampling unit reduces the resolution by using convolutional modules that control the stride. The backbone network combines the feature extraction unit and the downsampling unit into four stages. The first stage only has a feature extraction unit at the original resolution. The second stage uses the downsampling unit to add an additional 1 / 2 resolution feature extraction unit. Similarly, the third stage adds an additional 1 / 4 resolution feature extraction unit. The fourth stage only adds a branch starting with a 1 / 8 resolution feature map to connect to other modules.

4. The thyroid ultrasound image segmentation method based on a high-resolution neural network according to claim 2, characterized in that, The global modeling transformer module further includes a global modeling transformer weighted by x and y axis dimensional information, which establishes a global connection for the abstract feature map at 1 / 8 resolution.

5. The thyroid ultrasound image segmentation method based on a high-resolution neural network according to claim 2, characterized in that, The multi-scale fusion module includes an upsampling unit and a convolutional unit. The upsampling unit scales the feature maps with different resolutions generated after each stage in the backbone network to the same resolution, and then fuses the features at each scale through the convolutional unit.

6. The thyroid ultrasound image segmentation method based on a high-resolution neural network according to claim 2, characterized in that, The pyramid prediction module includes: a dilated convolution unit, a convolution unit, and a classifier unit. The dilated convolution unit performs dilated convolution processing of different sizes based on the four feature maps of different resolutions generated in the fourth stage to obtain a wider range of information connections. The convolution unit fuses the final result of the dilated convolution and outputs it to the classifier unit for binary classification to obtain a pixel-dimensional prediction map.

Images