A thyroid ultrasound image tumor segmentation classification and refined rating method

Abstract

The application discloses a thyroid ultrasound image tumor segmentation classification and refined rating method, which comprises constructing a thyroid nodule automatic segmentation classification model, and the model comprises a thyroid region prior guided feature enhancement network PGFE_Net, the thyroid region prior guided feature enhancement network PGFE_Net, an improved generative adversarial network GANs, a Resnet50 classification network and a gradient weighted class activation heat map Grad-CAM; wherein the PGFE_Net and the GANs extract the size, shape, edge characteristics and contrast information with the surrounding tissue of the nodule of the input image, generate a nodule mask, and perform a size prediction task; the nodule mask image and the original ultrasound image are combined, Resnet50 is used for feature discrimination and classification, accurate classification is carried out according to a TI-RADS standard, and the Grad-CAM provides visual interpretation of model decision-making. The application can effectively improve the detection and classification accuracy of thyroid nodule malignant tumors, realizes refined rating of the nodule, and provides auxiliary diagnosis support for clinicians.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing and medical image recognition, and particularly relates to a thyroid ultrasound image tumor segmentation classification and refined rating method. BACKGROUND

[0002] Thyroid nodules are common abnormal growths within the thyroid gland, which are usually identified through neck ultrasound imaging. These growths can be filled with solid or liquid, and their treatment is influenced by factors such as size and location. Current judgments on the thyroid are usually implemented by using the thyroid imaging reporting and data system (TI-RADS) to identify image information. The classification method of TI-RADS is to divide the thyroid nodules into different risk levels according to the characteristics of size, echo, edge, shape and calcification, so as to guide the clinician to decide whether biopsy or other further evaluation is needed.

[0003] With the continuous development of medical artificial intelligence technology, it has become a common means to use machine learning or deep learning to recognize ultrasound images and obtain corresponding reference information to assist medical personnel in diagnosis and treatment. In the aspect of TI-RADS, machine learning can supplement TI-RADS classification, especially when TI-RADS training data is scarce, the machine learning model can find the elements missed by TI-RADS and improve the detection of malignant tumors.

[0004] The prior art with publication number CN 113393512 A discloses a method and system for automatically estimating the volume of the thyroid based on two-dimensional ultrasound images, which is used to solve the technical problem of low efficiency of thyroid volume measurement and heavy workload of doctors caused by the high professional and difficult operation of traditional thyroid volume measurement. The publication number CN 115239655 A discloses a thyroid ultrasound image tumor segmentation and classification method and device, which is used to automatically mark the boundary of the tumor and provide benign and malignant classification results. CN 109350089 A discloses a thyroid region automatic segmentation method based on CT images, which is used for automatic segmentation of thyroid region. CN 114998296 A discloses a thyroid nodule segmentation method based on improved Unet network, which accurately locates the nodule region, enhances the global context and local information, and reduces the error of thyroid nodule segmentation. The article "Multi-Task Learning for Thyroid Nodule Segmentation with Thyroid Region Prio" published in the electronic paper https: / / blog.csdn.net / qq_40714949 / article / details / 123971916 discloses a thyroid region prior feature enhancement network for thyroid nodule segmentation, which enhances the segmentation of thyroid nodule. The above prior art has the following problems in using the thyroid ultrasound medical image segmentation and classification system:

[0005] 1. Since the shape, color and brightness of thyroid nodule are uncertain and may be confused with other organs, thyroid nodule is not easy to be automatically detected by the system, and the existing method ignores the echo and tissue composition factors, resulting in incorrect or missing automatic detection results, which hinders radiologists to diagnose in time, and even leads to incorrect thyroid biopsy position. Therefore, there is a lack of an accurate automatic detection method for thyroid ultrasound medical image processing and evaluation.

[0006] 2. The existing method can only make a basic judgment on the benign and malignant of thyroid nodule, and there is no detailed grading evaluation method and no stratified study on the malignant degree of nodule. Therefore, there is a lack of a method for evaluating thyroid nodule in detail.

[0007] Based on the above analysis, an improved thyroid imaging reporting and data system is needed to realize more accurate thyroid ultrasound image tumor segmentation and classification and precise rating. SUMMARY

[0008] 1. The technical problem to be solved is:

[0009] In view of the above technical problems, the present application provides a thyroid ultrasound image tumor segmentation classification and refined rating method, which not only realizes more accurate thyroid ultrasound image tumor segmentation classification, but also can perform refined rating.

[0010] 2. Technical solution:

[0011] A thyroid ultrasound image tumor segmentation classification and refined rating method, characterized by comprising the following steps:

[0012] Step one: build a thyroid nodule automatic segmentation classification model TN-ASC, which specifically includes a thyroid region prior guided feature enhancement network PGFE_Net, an improved generative adversarial network GANs, a Resnet50 classification network, and a gradient weighted class activation heat map Grad-CAM; wherein the PGFE_Net uses an encoder-decoder to extract the nodule size, shape and edge information in the nodule features and gland features of the input thyroid ultrasound image; the GANs extracts the echo and echo focus features of the input thyroid ultrasound image; the esnet50 classification network classifies and identifies the extracted features of the PGFE_Net and the GANs, and then inputs the gradient weighted class activation heat map Grad-CAM to generate a heat map, which displays the priority level to assist in diagnosis through the heat map;

[0013] Step two: build a sample data set for training the segmentation classification model; the samples in the sample data set are labeled with gland labels or nodule labels for thyroid ultrasound images, and then a sample set containing thyroid ultrasound images, gland images containing only gland labels, and nodule images containing only nodule labels is constructed;

[0014] Step three: train the segmentation classification model; set the training set and the test set according to the preset proportion to train the model and obtain the trained thyroid nodule automatic segmentation classification model;

[0015] Step four: input the thyroid ultrasound image to be evaluated into the trained model, and the nodule mask picture, gland mask picture and nodule size information output by the PGFE_Net, and the echo information output by the GANs are all sent into the Resnet50 classification network, and the Resnet50 classification network outputs features including nodule composition, echo, shape, edge definition, and echo focus;

[0016] Step 5: Design a thyroid nodule symptom scorer; This scorer is designed according to the TI-RADS classification standard for thyroid nodules. It includes multiple preset thyroid nodule-related features. Each feature is divided into multiple levels according to the severity of symptoms. Each level corresponds to a different score, thereby obtaining the scores of each level and the comprehensive score of the thyroid ultrasound image; and the image is classified according to the comprehensive score.

[0017] Step 6: Input the features contained in the scorer into the gradient-weighted class activation heatmap Grad-CAM according to their preset priority and score to generate a heatmap. The priority level is displayed through the heatmap to achieve auxiliary diagnosis.

[0018] Furthermore, the improved generative adversarial networks (GANs) are image synthesis networks based on Pix2Pix.

[0019] Furthermore, the thyroid region prior-guided feature enhancement network PGFE_Net specifically includes a 5-layer progressively connected shared encoder, a 4-layer progressively connected gland decoder, a 5-layer progressively connected nodule decoder, a nodule size prediction decoder, and an adaptive thyroid region prior guidance module ARPG. Each of the 1st to 4th layers of the shared encoder transmits the high-dimensional feature map extracted from the input image to the corresponding nodule decoder and gland decoder. The last layer of the shared encoder transmits its extracted features to the first layer's nodule decoder and gland decoder; simultaneously, the last layer's shared encoder transmits its extracted features to the nodule size prediction decoder. Each of the 1st to 3rd layers of the gland decoder is connected to the corresponding adaptive thyroid region prior guidance module ARPG to obtain prior information about the thyroid region and transmits this prior information to the corresponding layer's nodule decoder. Under the encoder's sharing mechanism, the last layer's gland decoder and nodule decoder output images containing the probability that each pixel belongs to a gland region or a nodule region. The nodule size prediction decoder outputs the predicted size of the nodules in the image.

[0020] Furthermore, the nodule size prediction decoder predicts the nodule size based on the feature map output by the fifth-layer shared encoder, and the predicted nodule size s pred As shown in the following formula:

[0021] s pred =sigmoid(MLP(f5))

[0022] Where sigmoid(·) represents the sigmoid(·) function, f5 represents the feature map of the fifth layer of the shared encoder in PGFE_Net, and MLP represents the multilayer perceptron network;

[0023] The Log-cosh loss function is used to calculate the loss L for the size prediction task of the nodule size prediction decoder. size As shown in the following formula:

[0024] Where: N nodule This indicates the number of thyroid nodule images input in this batch, s gt This represents the true area ratio of the nodule in the image. log(·) and cosh(·) represent the logarithmic function and the hyperbolic cosine function, respectively.

[0025] Furthermore, in step five, the Grad-CAM class activation heatmap uses the cross-entropy loss analysis of the thyroid nodule features as shown in the following formula to optimize the Grad-CAM parameters and improve the accuracy of visual display.

[0026]

[0027] In the above formula, The set representing the total loss; Indicates malignant / benign loss; This indicates component loss, which includes solid, mixed cystic, cystic, and spongy structures. Echo foci representing the loss of the comet's tail; Echoic foci indicating microcalcification loss; Echo foci indicating macroscopic loss of calcification; Echoic foci indicating loss of surrounding calcification; This indicates echo loss, which includes no echo, very low echo, low echo, and high echo. This indicates edge loss, which includes smoothing, unclear boundaries, irregularities, and extensions.

[0028] Furthermore, in the scoring device, the preset features related to thyroid nodules include nodule composition, echogenicity, current status, margin clarity, and echogenicity; the classification obtained based on the comprehensive score includes: TR1 benign, TR2 not suspicious, TR3 slightly suspicious, TR4 moderately suspicious, and TR5 severely suspicious.

[0029] Further, in step three, the model training specifically involves: inputting glandular image samples and nodule image samples corresponding to the same thyroid ultrasound image into PGFE_Net; PGFE_Net extracts the nodule size, shape, edge characteristics, and contrast information with surrounding tissues from the input image to generate glandular mask images and nodule mask images. The two types of mask images can represent the specific location and shape of thyroid nodules and glandular regions in the ultrasound image; inputting the ultrasound image into the GANs network to generate an image containing echo information; and combining the glandular mask image, nodule mask image, and echo information image with... The data is fed into a ResNet50 classification network. The ResNet50 network combines the features of all inputs to extract and classify various types of nodules. These various types of information include TI-RADS type, composition, echogenicity, edge, and echo focus. The type information output by the ResNet50 classification network is then input into Grad-CAM for training. Grad-CAM generates a coarse localization map of the basic region for each output type by examining the gradient of the last 2D feature map before the fully connected layer, thereby providing a visual explanation of the model's decisions and highlighting the regions that have the greatest impact on the model's decisions.

[0030] 3. Beneficial effects:

[0031] (1) The method disclosed in this paper is a thyroid ultrasound image tumor segmentation, classification and fine rating method, which extracts features from nodule information and glandular information respectively; the annotation of nodule and glandular information in thyroid ultrasound image is processed to obtain nodule image and glandular image respectively, and different images are batch-input into the corresponding feature extraction network for training when training the model, which can effectively improve the accuracy of feature extraction and thus help classify nodule types in the image.

[0032] (2) The TN-ASC automatic segmentation and classification model for thyroid nodules constructed in this method combines the PGFE_Net network, GANs network, ResNet50 model network, and Grad-CAM technology to ensure that the model can effectively improve the detection and classification accuracy of malignant thyroid nodules and achieve refined nodule rating.

[0033] (3) The method disclosed in this paper is a thyroid ultrasound imaging tumor segmentation, classification and refined rating method. The PGFE_Net network and GANs network used in this paper can effectively capture the size, shape, edge characteristics and contrast information with surrounding tissues of the nodules through the optimized feature extraction process, thereby enhancing the ability to analyze the characteristics of thyroid nodules.

[0034] (4) This invention discloses a method for segmentation, classification, and refined rating of thyroid ultrasound tumors. Utilizing Grad-CAM technology, this invention can provide a visual explanation of the model's decision-making process, highlighting the areas that have the greatest impact on the model's decisions. This visual explanation bridges the gap between deep learning models and expert radiologists' interpretations, providing doctors with a more intuitive basis for diagnosis. Attached Figure Description

[0035] Figure 1 This is an overall flowchart of a method for segmentation, classification, and refined rating of thyroid ultrasound tumors.

[0036] Figure 2 This is a diagram showing the overall structure of the TN-ASC automatic segmentation and classification model for thyroid nodules in this method.

[0037] Figure 3 This is a model diagram of the thyroid region prior-guided feature enhancement network PGFE_Net in this method;

[0038] Figure 4 This is a flowchart illustrating the nodule size prediction task implemented by the nodule size prediction decoder SPT in PGFE_Net within this method;

[0039] Figure 5 This is a flowchart of the adaptive region-leading ARPG module algorithm in PGFE_Net of this method;

[0040] Figure 6 This is a schematic diagram showing the overall input and output of the ResNet50 classification network in this method;

[0041] Figure 7 This is a flowchart illustrating the refined assessment of the thyroid nodule symptom scorer in this method.

[0042] Figure 8 This is a sample result diagram output by the ResNet50 classification network in specific embodiment 2;

[0043] Figure 9 This is a graph showing the accurate sampling results in specific embodiment 2;

[0044] Figure 10 This is a graph showing the results of inaccurate sampling in specific embodiment 2. Detailed Implementation

[0045] The present invention will now be described in detail with reference to the accompanying drawings.

[0046] As attached Figure 1 As shown, a method for segmentation, classification, and refined rating of thyroid ultrasound tumors is characterized by the following steps:

[0047] Step 1: Construct the TN-ASC automatic segmentation and classification model for thyroid nodules, specifically including the thyroid region prior-guided feature enhancement network PGFE_Net, improved generative adversarial networks (GANs), a ResNet50 classification network, and a gradient-weighted class activation heatmap (Grad-CAM). PGFE_Net uses an encoder and decoder to extract nodule features, nodule size and shape, and edge information from the glandular features of the input thyroid ultrasound image. GANs extract echo and echogenic foci features from the input thyroid ultrasound image. The ResNet50 classification network classifies and distinguishes the features extracted by PGFE_Net and GANs, then inputs them into the gradient-weighted class activation heatmap (Grad-CAM) to generate a heatmap. The heatmap displays priority levels to assist in diagnosis.

[0048] The automatic segmentation and classification model for thyroid nodules constructed in this invention is shown in the attached figure. Figure 2 As shown, the feature generation part of the model includes an upper branch of GANs networks and a lower branch of PGFE_Net networks. Glandular images are fed into the upper branch, while nodule images are fed into the lower branch for backpropagation of the loss. Simultaneously, nodule images are also fed into the upper branch to identify glandular regions. The obtained features are further fed into a region leader module to improve the accuracy of nodule region segmentation. A ResNet50 classification network serves as the feature discrimination part of the model, and then a heatmap is generated by setting a thyroid nodule symptom scorer.

[0049] Step 2: Construct a sample dataset for training the segmentation and classification model; the samples in the sample dataset are thyroid ultrasound images labeled with glandular labels or nodule labels, thereby constructing a sample set containing thyroid ultrasound images, glandular images containing only glandular labels, and nodule images containing only nodule labels;

[0050] Step 3: Train the segmentation and classification model; train the model using a training set and a test set according to a preset ratio to obtain a trained automatic segmentation and classification model for thyroid nodules;

[0051] Step 4: After inputting the thyroid ultrasound image to be evaluated into the trained model, the nodule mask image, gland mask image and nodule size information output by PGFE_Net, and the echo information output by GANs are all sent to the ResNet50 classification network. The ResNet50 classification network output includes features of nodule composition, echo, shape, edge clarity and echo foci.

[0052] Step 5: Design a thyroid nodule symptom scorer; This scorer is designed according to the TI-RADS classification standard for thyroid nodules. It includes multiple preset thyroid nodule-related features. Each feature is divided into multiple levels according to the severity of symptoms. Each level corresponds to a different score, thereby obtaining the scores of each level and the comprehensive score of the thyroid ultrasound image; and the image is classified according to the comprehensive score.

[0053] Step 6: Input the features contained in the scorer into the gradient-weighted class activation heatmap Grad-CAM according to their preset priority and score to generate a heatmap. The priority level is displayed through the heatmap to achieve auxiliary diagnosis.

[0054] Furthermore, the improved generative adversarial networks (GANs) are image synthesis networks based on Pix2Pix.

[0055] Furthermore, as shown in the appendix Figure 3 As shown, the thyroid region prior-guided feature enhancement network PGFE_Net specifically includes a 5-layer progressively connected shared encoder, a 4-layer progressively connected gland decoder, a 5-layer progressively connected nodule decoder, a nodule size prediction decoder, and an adaptive thyroid region prior guidance module ARPG. Each shared encoder in layers 1-4 transmits the high-dimensional feature map extracted from the input image to the corresponding nodule decoder and gland decoder. The last shared encoder transmits its extracted features to the first-layer nodule decoder and gland decoder; simultaneously, the last shared encoder transmits its extracted features to the nodule size prediction decoder. Each gland decoder in layers 1-3 is connected to the corresponding adaptive thyroid region prior guidance module ARPG to obtain prior information about the thyroid region and transmits this prior information to the corresponding nodule decoder. Under the encoder sharing mechanism, the last gland decoder and nodule decoder output images containing the probability that each pixel belongs to a gland region or a nodule region. The nodule size prediction decoder outputs the predicted size of the nodules in the image.

[0056] From the appendix Figure 3 As can be seen, the PGFE_Net network mainly consists of three parts: a shared encoder module, three independent decoders, and a 1-3 layer adaptive thyroid region prior guidance module. The encoder is designed to extract high-dimensional feature representations of the image. The three decoders are used for thyroid region segmentation, thyroid nodule detection, and nodule size prediction, respectively. The adaptive region prior guidance module is used to obtain prior information about the thyroid region to improve nodule segmentation performance. The encoder and decoder mentioned above can adopt the UNet vanilla encoder and decoder. The algorithm flowchart of the adaptive region prior guidance module is attached. Figure 5 As shown.

[0057] Furthermore, as shown in the appendix Figure 4 The nodule size prediction decoder shown predicts the nodule size based on the feature map output by the fifth layer shared encoder. The predicted nodule size s pred As shown in the following formula:

[0058] s pred =sigmoid(MLP(f5))

[0059] Where sigmoid(·) represents the sigmoid(·) function, f5 represents the feature map of the fifth layer of the shared encoder in PGFE_Net, and MLP represents the multilayer perceptron network;

[0060] The Log-cosh loss function is used to calculate the loss L for the size prediction task of the nodule size prediction decoder. size As shown in the following formula:

[0061] Where: N nodule This indicates the number of thyroid nodule images input in this batch, s gt This represents the true area ratio of the nodule in the image. log(·) and cosh(·) represent the logarithmic function and the hyperbolic cosine function, respectively.

[0062] As attached Figure 4 The flowchart shown is for the nodule size prediction task. Nodule size prediction is performed by the size prediction task (SPT). After the ultrasound image enters the encoder, it undergoes global average pooling processing, and the predicted nodule size is obtained using the formula described above. Furthermore, in step five, the Grad-CAM class activation heatmap uses the cross-entropy loss analysis formula to determine the corresponding loss of thyroid nodule features, thereby optimizing Grad-CAM parameters and improving the accuracy of visual display.

[0063]

[0064] In the above formula, The set representing the total loss; Indicates malignant / benign loss; This indicates component loss, which includes solid, mixed cystic, cystic, and spongy structures. Echo foci representing the loss of the comet's tail; Echoic foci indicating microcalcification loss; Echo foci indicating macroscopic loss of calcification; Echoic foci indicating loss of surrounding calcification; This indicates echo loss, which includes no echo, very low echo, low echo, and high echo. This indicates edge loss, which includes smoothing, unclear boundaries, irregularities, and extensions.

[0065] Furthermore, in the scoring device, the preset features related to thyroid nodules include nodule composition, echogenicity, current status, margin clarity, and echogenicity; the classification obtained based on the comprehensive score includes: TR1 benign, TR2 not suspicious, TR3 slightly suspicious, TR4 moderately suspicious, and TR5 severely suspicious.

[0066] By setting up a thyroid nodule symptom scoring system, as shown in the attached document... Figure 7 As shown, this enables more refined evaluation.

[0067] Further, in step three, the model training specifically involves: inputting glandular image samples and nodule image samples corresponding to the same thyroid ultrasound image into PGFE_Net; PGFE_Net extracts the nodule size, shape, edge characteristics, and contrast information with surrounding tissues from the input image to generate glandular mask images and nodule mask images. The two types of mask images can represent the specific location and shape of thyroid nodules and glandular regions in the ultrasound image; inputting the ultrasound image into the GANs network to generate an image containing echo information; and combining the glandular mask image, nodule mask image, and echo information image with... The data is fed into a ResNet50 classification network. The ResNet50 network combines the features of all inputs to extract and classify various types of nodules. These various types of information include TI-RADS type, composition, echogenicity, edge, and echo focus. The type information output by the ResNet50 classification network is then input into Grad-CAM for training. Grad-CAM generates a coarse localization map of the basic region for each output type by examining the gradient of the last 2D feature map before the fully connected layer, thereby providing a visual explanation of the model's decisions and highlighting the regions that have the greatest impact on the model's decisions. Specific Implementation Example 1:

[0069] This embodiment uses the TN3K and TG3K thyroid nodule region segmentation datasets to train the TN-ASC model. The TN3K dataset is a dataset formed by preprocessing ultrasound images from various ultrasound imaging systems, including nodule labeling, annotation confirmation, and thyroid nodule image verification. It is divided into a training set and a test set, with the training set containing 2879 images and the test set containing 614 images. The TG3K dataset consists of 16 ultrasound videos proposed by Wunderling et al. Frames were extracted from these videos to construct the thyroid imaging segmentation dataset, which contains a total of 3585 thyroid ultrasound images.

[0070] First, one image is selected from each of the TN3K and TG3K datasets. These images are then labeled as nodules and glands respectively, resulting in a mini-batch of images. The gland images are fed into the upper branch, while the nodule images are fed into the lower branch for backpropagation of the loss. Simultaneously, the nodule images are also fed into the upper branch to identify gland regions. The obtained features are further fed into the region leader module to improve the accuracy of nodule region segmentation. Finally, to achieve the goal of nodule segmentation, as shown in the appendix... Figure 5 The adaptive region-leading ARPG shown adds a refinement layer after the decoder of the nodule segmentation branch.

[0071] The model was trained by inputting glandular and nodule image samples into the PGFE_Net network. The PGFE_Net network includes a shared encoder, three independent decoders, and an Adaptive Thyroid Region Prior Guided Module (ARPG) to extract nodule size, shape, edge characteristics, and contrast information with surrounding tissue from the input images. Glandular and nodule mask images were generated using PGFE_Net. These mask images represent the specific location and shape of thyroid nodules and glandular regions in the ultrasound image. The same thyroid ultrasound image was then input into a GAN network to generate an image containing echo information. The GAN network, using a Conditional GANs structure, extracted the echo characteristics of the image. The glandular mask image, nodule mask image, and echo information image were simultaneously fed into a ResNet50 classification network. The ResNet50 network combined these input features to extract and classify various types of nodule information, including TI-RADS type, composition, echogenicity, edges, and echo focus. The type information output by the ResNet50 classification network is fed into Grad-CAM for training. Grad-CAM generates coarse localization maps of the basic regions for each output type by examining the gradient of the last 2D feature map before the fully connected layers, thus providing a visual interpretation of the model's decisions and highlighting the regions that have the greatest impact on the model's decisions. Specific Implementation Example 2:

[0073] Reference Figures 8-10 This is an embodiment 2 of the present invention. To verify the beneficial effects of the present invention, a test set was obtained from the TN3K and TG3K datasets to test the classification accuracy of the TN-ASC automatic segmentation and classification model for thyroid nodules constructed in this method, and the experimental results were used for scientific demonstration.

[0074] The test set is input into the classification and segmentation part of the trained model, namely: the thyroid region prior guided feature enhancement network PGFE_Net, improved generative adversarial networks (GANs), and the ResNet50 classification network; and the information on thyroid nodule-related symptoms that can be identified by the ResNet50 classification network on the input test set is obtained, as shown in Table 1:

[0075] Table 1. Statistical table of accuracy of nodule malignancy classification and TI-RADS classification.

[0076]

[0077]

[0078] After visualizing the test set using Grad-CAM, this method enables separate analysis of each category and seeks the expertise of radiology specialists to interpret the results. Figure 7 and Figure 8 This example demonstrates accurate result samples and corresponding expert explanations. Figure 9 Showing samples of erroneous results and expert explanations of these situations.

[0079] like Figure 7 As shown, benign lesions exhibit a clear mixed solid-cystic component and are classified as TI-RADS-2, thus indicating a low suspicion of malignancy (1.5%). The thyroid ultrasound medical image processing and evaluation / assisted medical system model based on the TN-ASC model accurately identifies the mixed solid-cystic component, hyperechoic solid region, and smooth boundary, all of which contribute to the high probability of a benign diagnosis.

[0080] like Figure 8 As shown, the TI-RADS-5 malignant nodule was characterized by hypoechoic components, lobulated margins, internal microcalcifications, and a higher-than-wide appearance, all of which indicated a high suspicion of malignancy (35%). Subsequent pathological confirmation was performed. The thyroid ultrasound medical image processing and evaluation / assisted medical system model based on the TN-ASC model accurately identified the areas with microcalcifications and irregular boundaries.

[0081] The following conclusions can be drawn from the analysis of this specific embodiment:

[0082] This method utilizes artificial intelligence analysis to simulate the cognitive process of radiologists, specifically assessing the differences between this method and the fundamental principles outlined in the ACR TI-RADS guidelines. The final Grad-CAM heatmap shows that the method's analytical capabilities are largely consistent with radiologists' reasoning. However, experiments revealed that this invention cannot distinguish between microcalcifications and macrocalcifications—a distinction typically made based on calcification size.

[0083] Furthermore, in the case of pure cystic nodules, while this method successfully identifies both cystic and solid components in mixed cystic cases, it exhibits uncertainty in the interpretation of anechoic structures. When applying this method to the TI-RADS classification task, the following factors should be considered:

[0084] 1. Identify and integrate the presence of cystic and / or solid components within the nodule to recalculate its composition;

[0085] 2. Adjust the echo model to generate regression output to facilitate the measurement of nodule contrast;

[0086] 3. Unclear edges may be ignored because the model is able to identify nodule boundaries more accurately than the human eye.

[0087] like Figure 9 As shown, the malignant lesion, characterized by well-defined isoechoic to very hypoechoic solid components and internal microcalcifications, is classified as TI-RADS-5, indicating a high suspicion of malignancy (35%). It can be seen that the thyroid ultrasound medical image processing and evaluation / assisted medical system model for thyroid nodules constructed in this application incorrectly indicates a high probability of benignity, fails to detect very hypoechoic components, and cannot detect internal punctate echogenic foci. Specific Implementation Example 3

[0089] To address the issues identified in Example 2, Grad-CAM technology was used to further analyze the model. Based on the scoring system and the Grad-CAM analysis, the output of the neural network was modified.

[0090] In the initial version of the model, the output layer of the composition is a four-unit layer with softmax, corresponding to capsule-like compositions, solid compositions, mixed compositions, or sponge-like compositions.

[0091] Next, the four outputs were replaced with three independent logical sigmoid outputs corresponding to cystic, solid, and spongy shapes. When both cystic and solid outputs were above a threshold of 0.5, the outputs were programmatically mixed. The echo output layer initially had four softmax units, which were replaced with single linear outputs with target values ​​of 0, 1, 2, and 3, encoding high echo, low echo, very low echo, and no echo categories. Inference was considered correct if the nearest integer in the output was the target class. Finally, the margin output units corresponding to ambiguous shape classes were removed and merged with the "smooth" shape margins. This is clinically reasonable, as the TI-RADS scores for these four edge types are identical. This simplified model showed enhanced results after training, as shown in Table 2.

[0092] Table 2. Results of model enhancement using Grad-CAM

[0093]

[0094]

[0095] The results of this experiment provide strong evidence of consistency between automatic classification in accordance with the ACR TI-RADS guidelines and popular radiological theories.

[0096] The combined results of specific embodiments 2 and 3 demonstrate that this method can provide important assistance to doctors and radiologists in accurately detecting and classifying ACRTI-RADS and malignant tumors.

[0097] Although the present invention has been disclosed above with reference to preferred embodiments, these are not intended to limit the invention. Any person skilled in the art can make various changes or modifications without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention should be defined by the scope of the claims of this application.

Claims

1. A method for segmentation, classification, and refined rating of thyroid ultrasound tumors, characterized in that: Includes the following steps: Step 1: Construct the TN-ASC automatic segmentation and classification model for thyroid nodules, which specifically includes the thyroid region prior-guided feature enhancement network PGFE_Net, improved generative adversarial networks (GANs), a ResNet50 classification network, and a gradient-weighted class activation heatmap (Grad-CAM). PGFE_Net uses an encoder and decoder to extract nodule features, nodule size and shape, and edge information from the glandular features of the input thyroid ultrasound image. GANs extract echo and echo foci features from the input thyroid ultrasound image. The ResNet50 classification network classifies and distinguishes the features extracted by PGFE_Net and GANs, and then inputs them into a gradient-weighted class activation heatmap (Grad-CAM) to generate a heatmap. The heatmap displays the priority level to assist in diagnosis. Step 2: Construct a sample dataset for training the segmentation and classification model; the samples in the sample dataset are thyroid ultrasound images labeled with glandular labels or nodule labels, thereby constructing a sample set containing thyroid ultrasound images, glandular images containing only glandular labels, and nodule images containing only nodule labels; Step 3: Train the segmentation and classification model; train the model using a training set and a test set according to a preset ratio to obtain a trained automatic segmentation and classification model for thyroid nodules; Step 4: After inputting the thyroid ultrasound image to be evaluated into the trained model, the nodule mask image, gland mask image and nodule size information output by PGFE_Net, and the echo information output by GANs are all sent to the ResNet50 classification network. The ResNet50 classification network output includes features of nodule composition, echo, shape, edge clarity and echo foci. Step 5: Design a thyroid nodule symptom scorer; This scorer is designed according to the TI-RADS classification standard for thyroid nodules. It includes multiple preset thyroid nodule-related features. Each feature is divided into multiple levels according to the severity of symptoms. Each level corresponds to a different score, thereby obtaining the scores of each level and the comprehensive score of the thyroid ultrasound image. The images are then categorized based on their overall scores. Step 6: Input the features contained in the scorer into the gradient-weighted class activation heatmap Grad-CAM according to their preset priority and score to generate a heatmap. The priority level is displayed through the heatmap to achieve auxiliary diagnosis.

2. The method for segmentation, classification, and refined rating of thyroid ultrasound tumors according to claim 1, characterized in that: The improved generative adversarial networks (GANs) are image synthesis networks based on Pix2Pix.

3. The method for segmentation, classification, and refined rating of thyroid ultrasound tumors according to claim 2, characterized in that: The thyroid region prior-guided feature enhancement network PGFE_Net specifically includes a 5-layer progressively connected shared encoder, a 4-layer progressively connected gland decoder, a 5-layer progressively connected nodule decoder, a nodule size prediction decoder, and an adaptive thyroid region prior-guided module ARPG. Each layer of the shared encoder in layers 1-4 passes the high-dimensional feature map extracted from the input image to the nodule decoder and gland decoder of the corresponding layer; the shared encoder of the last layer passes its extracted features to the nodule decoder and gland decoder of the first layer; at the same time, the shared encoder of the last layer passes its extracted features to the nodule size prediction decoder; the gland decoder of each layer in layers 1-3 is connected to the corresponding adaptive thyroid region prior guidance module ARPG to obtain the prior information of the thyroid region and passes the prior information to the nodule decoder of the corresponding layer; the image output by the gland decoder and nodule decoder of the last layer under the encoder sharing mechanism includes the probability that each pixel belongs to the gland region or the nodule region; the nodule size prediction decoder outputs the predicted size of the nodules in the image.

4. The method for segmentation, classification, and refined rating of thyroid ultrasound tumors according to claim 3, characterized in that: The nodule size prediction decoder predicts the nodule size based on the feature map output by the fifth-layer shared encoder, and the predicted nodule size s pred As shown in the following formula: s pred =sigmoid(MLP(f5)) Where sigmoid(·) represents the sigmoid(·) function, f5 represents the feature map of the fifth layer of the shared encoder in PGFE_Net, and MLP represents the multilayer perceptron network; The Log-cosh loss function is used to calculate the loss L for the size prediction task of the nodule size prediction decoder. soze As shown in the following formula: Where: N nodule This indicates the number of thyroid nodule images input in this batch, s gt This represents the true area ratio of the nodule in the image; log(·) and cosh(·) represent the logarithmic function and the hyperbolic cosine function, respectively.

5. The method for segmentation, classification, and refined rating of thyroid ultrasound tumors according to claim 4, characterized in that: In step five, the Grad-CAM class activation heatmap uses the cross-entropy loss analysis of the thyroid nodule features as shown in the following formula to optimize Grad-CAM parameters and improve the accuracy of visual display. In the above formula, The set representing the total loss; Indicates malignant / benign loss; This indicates component loss, which includes solid, mixed cystic, cystic, and spongy structures. Echo foci representing the loss of the comet's tail; Echoic foci indicating microcalcification loss; Echo foci indicating macroscopic loss of calcification; Echoic foci indicating the loss of surrounding calcification; This indicates echo loss, which includes no echo, very low echo, low echo, and high echo. This indicates edge loss, which includes smoothing, unclear boundaries, irregularities, and extensions.

6. The method for segmentation, classification, and refined rating of thyroid ultrasound tumors according to claim 5, characterized in that: The scoring system includes preset features related to thyroid nodules such as nodule composition, echogenicity, current status, margin clarity, and echogenicity. The classification based on the comprehensive score includes: TR1 benign, TR2 not suspicious, TR3 slightly suspicious, TR4 moderately suspicious, and TR5 severely suspicious.

7. The method for segmentation, classification, and refined rating of thyroid ultrasound tumors according to claim 6, characterized in that: The specific steps of training the model in step three are as follows: Input the glandular image sample and nodule image sample corresponding to the same thyroid ultrasound image into PGFE_Net; PGFE_Net extracts the nodule size, shape, edge characteristics, and contrast information with surrounding tissues from the input image to generate glandular mask images and nodule mask images. The two types of mask images can represent the specific location and shape of thyroid nodules and glandular regions in the ultrasound image; input the ultrasound image into the GANs network to generate an image containing echo information; simultaneously send the glandular mask image, nodule mask image, and echo information image to... The ResNet50 classification network is used to extract and classify various types of nodules by combining features from all inputs. These various types of information include TI-RADS type, composition, echogenicity, edge, and echo focus. The type information output by the ResNet50 classification network is then input into Grad-CAM for training. Grad-CAM generates a coarse localization map of the basic region for each output type by examining the gradient of the last 2D feature map before the fully connected layer, thereby providing a visual explanation of the model's decisions and highlighting the regions that have the greatest impact on the model's decisions.

Images