Methods, systems, and storage media for constructing thyroid nodule auxiliary diagnostic models based on multimodal data.

Abstract

This application provides a method, system, and storage medium for constructing an auxiliary diagnostic model for thyroid nodules based on multimodal data. The method includes: acquiring multimodal thyroid nodule medical data; classifying the thyroid nodule medical data of each modality using a unimodal classification model to obtain intermediate classification results for each thyroid nodule corresponding to the medical data of each modality; and weightedly fusing all intermediate classification results to obtain a benign / malignant classification result for thyroid nodules based on the multimodal thyroid nodule medical data. This application uses a unimodal classification model to classify medical data of each modality, eliminating the step of converting multimodal data into structured data, thus improving data processing efficiency. This application can also be used for the diagnosis, monitoring, and treatment of thyroid nodules, assisting clinicians in improving the diagnostic efficiency of thyroid nodules and reducing the clinical burden.

Description

Technical Field

[0001] This application relates to the fields of medical image processing and computer-aided disease diagnosis, specifically to a method, system, and storage medium for constructing an auxiliary diagnostic model for thyroid nodules based on multimodal data. Background Technology

[0002] The Bethesda classification is the standard reporting system for fine-needle aspiration cytology (FNA) of the thyroid gland. It classifies thyroid nodules (TN) into grades I-VI based on their malignancy risk and clinical management recommendations. Grade I thyroid nodules (undiagnostic / unsatisfactory specimens) cannot be diagnosed due to insufficient cellularity or interfering factors, requiring repeat aspiration or ultrasound evaluation; malignancy risk is 1%-4%. Grade II thyroid nodules (benign lesions) are recommended for regular ultrasound follow-up. Grade III thyroid nodules (atypical lesions of indeterminate significance) have cytological features between benign and malignant, with a malignancy risk of 10%-30%, requiring molecular testing or repeat aspiration. Grade IV thyroid nodules (follicular tumors / suspicious follicular tumors) have a malignancy risk of 15%-40%, and surgical resection is usually recommended for definitive diagnosis. Grade V thyroid nodules (suspicious malignant tumors) show significant nuclear atypia, with a malignancy risk of 60%-75%, requiring surgical resection and intraoperative pathological examination. Grade VI thyroid nodules (malignant tumors), diagnosed as thyroid cancer (such as papillary carcinoma), have a malignancy risk of 97%-99%. Surgical plans must be developed based on the pathological type.

[0003] However, due to the "uncertainty" of Bethesda Class III TN (the inability to directly determine benignity or malignancy), there is no standard treatment plan in clinical practice. Doctors' recommendations are very broad, potentially including repeat biopsies, molecular testing, diagnostic resection, and follow-up monitoring. However, practice shows that even with repeat biopsies, 10-30% of cytological results are still Bethesda Class III; molecular testing has low sensitivity, with a 59% probability of malignancy for atypical lesions, and negative results are unreliable, while positive results do not change treatment recommendations, and are expensive with low cost-effectiveness; diagnostic lobectomy may lead to unnecessary surgery, and approximately 2-10% of patients may experience serious surgical complications (hoarseness, hypocalcemia, etc.), forcing most patients to undergo lifelong daily levothyroxine replacement therapy; follow-up monitoring may lead to missed diagnoses and delayed treatment for some patients. Therefore, there is an urgent need to find a method for constructing an auxiliary diagnostic model for thyroid nodules, a diagnostic method, and a system to accurately differentiate between benign and malignant Bethesda Class III TN, reducing unnecessary surgeries and missed tumor diagnoses. Summary of the Invention

[0004] This application is made in view of at least one of the aforementioned technical problems existing in conventional technologies. According to one aspect of this application, a method for constructing an auxiliary diagnostic model for thyroid nodules based on multimodal data is provided, the method comprising:

[0005] Acquire multimodal thyroid nodule medical data; wherein, the multimodal thyroid nodule medical data includes clinical data and image data;

[0006] The medical data of thyroid nodules in each modality were classified using a single-modality classification model to obtain the intermediate classification results of each thyroid nodule corresponding to the medical data of each modality.

[0007] The intermediate classification results of all thyroid nodules were weighted and fused to obtain the benign and malignant classification results of thyroid nodules based on multimodal thyroid nodule medical data.

[0008] In some embodiments, where the thyroid nodule medical data includes ultrasound images and cytological images, the method further includes:

[0009] Perform data augmentation on image data samples to obtain multiple intermediate image data samples;

[0010] At least two different image classification networks are constructed as the candidate first classification model. The collected medical image samples and their corresponding classification labels are used as the input of the candidate first classification model. During the training process, based on the transfer learning strategy, the pre-trained image feature extraction network is used as the base model to extract features from the input image and obtain feature vectors. Then, these vectors are input into the constructed fully connected layer classifier for feature processing and classification.

[0011] Five-fold cross-validation was used to validate at least two candidate first-class classification models to obtain the accuracy data of each candidate first-class classification model.

[0012] Based on the accuracy data of each candidate first classification model, the model with the best area under the curve performance is selected from at least two candidate first classification models as the first classification model, so that the first classification model can classify thyroid nodules as benign or malignant based on the image data.

[0013] In some embodiments, data augmentation operations are performed on the plurality of image data samples to obtain a plurality of intermediate image data samples, including:

[0014] Each image data sample is cropped to convert it into an image of a preset size.

[0015] The image of the preset size is standardized, wherein the standardization operation is based on the standard values ​​of the pre-trained model;

[0016] The standardized image data samples are subjected to a preset number of random rotation operations to obtain multiple intermediate image data samples at different angles.

[0017] In some embodiments, a five-fold cross-validation method is used to validate at least two candidate first-class classification models to obtain accuracy data for each candidate first-class classification model, including:

[0018] For any candidate first classification model, the plurality of intermediate image data samples are divided into five mutually exclusive subsets;

[0019] One subset was used as the validation set, and the other four subsets were used as the training set. Five training iterations and five validation iterations were performed using the Adam optimization algorithm and the cross-entropy loss function to obtain five performance scores.

[0020] The performance of each candidate first classification model is evaluated based on the area under the corresponding receiver operating characteristic curve. The average of the five performance scores is calculated as the performance evaluation data of the candidate first classification model.

[0021] The best-performing model among the candidate first classification models is selected as the first classification model, which then classifies thyroid nodules as benign or malignant based on the image data.

[0022] In some embodiments,

[0023] The image data includes ultrasound images or cytological images;

[0024] The first classification model includes a first ultrasound classification model or a first cell classification model; wherein, the first ultrasound classification model is used to classify ultrasound images to obtain a first intermediate classification result; and the first cell classification model is used to classify cytological images to obtain a second intermediate classification result.

[0025] In some embodiments, where the thyroid nodule medical data includes clinical data, the method further includes:

[0026] Acquire multiple clinical data samples;

[0027] Data preprocessing is performed on the multiple clinical data samples, including: quantizing and encoding categorical attributes, deleting features with missing values ​​exceeding a preset proportion, and performing maximum-min normalization on numerical features to improve the stability of model training.

[0028] Based on multiple clinical data after data preprocessing, each sample is represented as a feature vector and input into at least two different machine learning algorithms for modeling training to obtain at least two candidate secondary classification models;

[0029] Five-fold cross-validation was used to train and validate the candidate secondary classification model. During the training process, stochastic gradient descent was used to iteratively update the model parameters, and grid search was used for tuning.

[0030] The model with the best area under the curve performance among the at least two candidate second classification models is selected as the second classification model. Based on the clinical data, the second classification model classifies thyroid nodules into benign and malignant types, and obtains a third intermediate classification result.

[0031] In some embodiments, selecting the model with the best area under the curve performance from the at least two candidate second classification models as the second classification model includes:

[0032] Receiver operating characteristic curves are generated based on the probability values ​​of the classification results of each candidate secondary classification model, and the corresponding area under the curve is calculated to evaluate the performance of each candidate secondary classification model.

[0033] The candidate second classification model with the best area under the curve performance was selected as the second classification model.

[0034] In some embodiments, all intermediate classification results of thyroid nodules are weighted and fused to obtain benign and malignant classification results of thyroid nodules based on multimodal thyroid nodule medical data, including:

[0035] Receive the first intermediate classification result output by the first ultrasound classification model;

[0036] Receive the second intermediate classification result output by the first cell classification model;

[0037] Receive the third intermediate classification result output by the second classification model;

[0038] The first intermediate classification result, the second intermediate classification result, and the third intermediate classification result are weighted and fused to obtain multiple sets of intermediate benign and malignant classification results;

[0039] The result with the best area under the curve from the multiple intermediate benign and malignant classification results is selected as the benign and malignant classification result of the thyroid nodule.

[0040] In some embodiments, the first weight includes the area under the curve performance value of the first ultrasound classification model;

[0041] The second weight includes the area under the curve performance value of the first cell classification model;

[0042] The third weight includes the area under the curve performance value of the second classification model.

[0043] In some embodiments, the ultrasound image includes at least one of the following: ultrasound elastography, grayscale ultrasound image, and / or color Doppler ultrasound image;

[0044] The clinical data include at least one of the following: gender, age, height, weight, past medical history, duration of thyroid nodule discovery, growth rate of thyroid nodule, and Bethesda classification of thyroid nodule.

[0045] According to another aspect of this application, embodiments of this application also provide a thyroid nodule auxiliary diagnostic system based on multimodal data, the system comprising:

[0046] The acquisition module is used to acquire multimodal thyroid nodule medical data from at least one patient; the multimodal thyroid nodule medical data includes clinical data and image data;

[0047] The intermediate classification module is used to classify the thyroid nodule medical data of each patient in each modality using a single-modality classification model, and obtain the intermediate classification results of each thyroid nodule corresponding to the thyroid nodule medical data of each patient in each modality.

[0048] The weighted fusion module is used to weight and fuse the intermediate classification results of all thyroid nodules for each patient to obtain the benign or malignant classification result of the thyroid nodules for each patient.

[0049] In some embodiments, the image data includes ultrasound images and cytological images; the intermediate classification module includes a first classification submodule, a second classification submodule, and a third classification submodule; wherein,

[0050] The first classification submodule is used to classify the ultrasound image using a first ultrasound classification model to obtain a first intermediate classification result;

[0051] The second classification submodule is used to classify the cytology image using the first cell classification model to obtain a second intermediate classification result;

[0052] The third classification submodule is used to classify the clinical data using the second classification model to obtain a third intermediate classification result.

[0053] In some embodiments, wherein

[0054] The first ultrasound classification model is constructed based on the following algorithms: 50-layer residual network algorithm, DenseNet121 algorithm, GoogleNet algorithm, Vision Transformer algorithm, Vision Transformer algorithm, convolutional neural network algorithm, Swing Transformer algorithm, VGG19 algorithm, or InceptionV3 algorithm.

[0055] The first cell classification model is constructed based on the following algorithms: 50-layer residual network algorithm, DenseNet121 algorithm, GoogleNet algorithm, Vision Transformer algorithm, convolutional neural network algorithm, 50-layer residual network algorithm, VGG19 algorithm, InceptionV3 algorithm, or Swing Transformer algorithm.

[0056] The second classification model is built based on the following algorithms: logistic regression, support vector classification, K-nearest neighbors, random forest, or extreme gradient boosting.

[0057] In some embodiments, the weighted fusion module is specifically used to calculate the first sum of the product of the first intermediate classification result and the first weight, the product of the second intermediate classification result and the second weight, and the product of the third intermediate classification result and the third weight, and to calculate the second sum of the first weight, the second weight, and the third weight, and divide the first sum by the second sum to obtain the benign or malignant classification result of the thyroid nodules for each patient.

[0058] In some embodiments, wherein

[0059] The first weight includes the area under the curve performance value of the first ultrasound classification model;

[0060] The second weight includes the area under the curve performance value of the first cell classification model;

[0061] The third weight includes the area under the curve performance value of the second classification model.

[0062] In some embodiments, the ultrasound image includes at least one of the following: ultrasound elastography, grayscale ultrasound image, and color Doppler ultrasound image;

[0063] The clinical data include at least one of the following: sex, age, height, weight, past medical history, duration of thyroid nodule discovery, thyroid nodule growth rate, and Bethesda classification of the thyroid nodule.

[0064] According to another aspect of this application, embodiments of this application also provide a thyroid nodule auxiliary diagnostic terminal based on multimodal data, the terminal comprising:

[0065] The system includes a memory and a processor, wherein the memory stores a computer program that is executed by the processor, and when the computer program is executed by the processor, the processor causes the processor to perform the thyroid nodule auxiliary diagnostic model construction method based on multimodal data as described above.

[0066] According to another aspect of this application, embodiments of this application also provide a storage medium, wherein the storage medium includes: a computer program stored on the storage medium, the computer program causing the processor to execute the thyroid nodule auxiliary diagnostic model construction method based on multimodal data as described above when executed by a processor.

[0067] According to another aspect of this application, embodiments of this application also provide a computer program product, wherein the computer program / instructions, when executed by a processor, implement the thyroid nodule auxiliary diagnostic model construction method based on multimodal data as described above.

[0068] Compared with traditional technologies, the beneficial effects of this application are as follows:

[0069] The method for constructing an auxiliary diagnostic model for thyroid nodules based on multimodal data in this application involves acquiring multimodal clinical data and / or image data of thyroid nodules, classifying them using a single-modal classification model corresponding to the clinical data and / or image data, obtaining intermediate classification results for thyroid nodules output by each single-modal classification model, and then weighted and fused the intermediate classification results output by all single-modal classification models to obtain a benign or malignant classification result of thyroid nodules based on multimodal thyroid nodule medical data. This application uses a single-modal classification model to classify medical data of each modality and fuses the results output by each single-modal classification model, eliminating the step of uniformly converting multimodal data into structured data. Therefore, it is more efficient and has higher scenario adaptability in the data processing stage. Furthermore, this application can diagnose, monitor, and treat Bethesda III thyroid nodules, assisting clinicians in improving the diagnostic efficiency of thyroid nodules, reducing the clinical burden, and providing important theoretical and technical support for the early identification and individualized treatment of benign and malignant Bethesda III thyroid nodules.

[0070] Furthermore, the thyroid nodule auxiliary diagnostic system based on multimodal data in this application embodiment can process multimodal thyroid nodule medical data from multiple patients simultaneously. When classifying multimodal thyroid nodule medical data, it classifies the thyroid nodule medical data of each patient for each modality, and then fuses the intermediate classification results to assist clinicians in improving the diagnostic efficiency of thyroid nodules, reducing the clinical burden, and providing important theoretical and technical support for the early identification and treatment of benign and malignant Bethesda III thyroid nodules. Attached Figure Description

[0071] Figure 1 A schematic flowchart illustrating a method 100 for constructing an auxiliary diagnostic model for thyroid nodules based on multimodal data according to an embodiment of this application is shown.

[0072] Figure 2 A schematic flowchart illustrating a process 200 for training a first classification model based on image data according to an embodiment of this application is shown.

[0073] Figure 3 A schematic flowchart illustrating step S201 according to an embodiment of this application is shown;

[0074] Figure 4 A schematic flowchart illustrating step S204 according to an embodiment of this application is shown;

[0075] Figure 5 A schematic flowchart illustrating step S205 according to an embodiment of this application is shown;

[0076] Figure 6 A schematic flowchart illustrating a process 600 of training a second classification model based on clinical data according to an embodiment of this application is shown.

[0077] Figure 7 A schematic flowchart of step S605 according to an embodiment of this application is shown;

[0078] Figure 8 A schematic flowchart illustrating step S103 according to an embodiment of this application is shown;

[0079] Figure 9 A schematic flowchart illustrating a method 900 for constructing an auxiliary diagnostic model for thyroid nodules according to another embodiment of this application is shown.

[0080] Figure 10 A schematic flowchart of a thyroid nodule auxiliary diagnosis method 1000 according to an embodiment of this application is shown;

[0081] Figure 11 A schematic block diagram of a thyroid nodule auxiliary diagnostic system 1100 based on multimodal data according to an embodiment of this application is shown.

[0082] Figure 12 A schematic block diagram of an intermediate classification module 1102 according to an embodiment of this application is shown;

[0083] Figure 13 A schematic block diagram of a thyroid nodule auxiliary diagnostic terminal 1300 based on multimodal data according to an embodiment of this application is shown. Detailed Implementation

[0084] The present application is further illustrated below with reference to embodiments. It should be understood that the embodiments are only used to further illustrate and explain the present application and are not intended to limit the present application.

[0085] Unless otherwise defined, technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art. While similar or identical methods and materials may be applied in experimental or practical applications, materials and methods are described herein. In case of conflict, the definitions included herein shall prevail. Furthermore, materials, methods, and examples are for illustrative purposes only and are not intended to be limiting. The present application is further described below with reference to specific embodiments, but is not intended to limit the scope of the application.

[0086] Thyroid nodules (TN) are lumps within the thyroid gland that move up and down with the thyroid during swallowing. They are a common clinical condition caused by various factors. In recent years, the incidence of thyroid tumors has increased significantly, especially in some coastal cities, where the incidence of malignant tumors is rising at an average annual rate of 4%, making it one of the fastest-growing malignant tumors. The current incidence of thyroid nodules in China is 12.8-18.6%, of which 5-15% are malignant. Different types of nodules require different treatments. Malignant thyroid lesions are primarily treated with surgical removal of the lesion and part or all of the thyroid tissue, supplemented by radiotherapy. Benign thyroid lesions, on the other hand, may require non-surgical, selective surgery, or targeted treatments such as regular follow-up. Early identification of the nature of thyroid nodules can not only alleviate patient anxiety, improve quality of life, and improve prognosis, but also prevent over-treatment, unnecessary waste of medical resources, and reduce the patient's financial burden. Therefore, the most important task after the discovery of a thyroid nodule is to differentiate between benign and malignant nodules.

[0087] Ultrasound examination is the most commonly used method for examining thyroid nodules in clinical practice, offering advantages such as safety, non-invasiveness, and low cost. For example, benign and malignant thyroid nodules exhibit different characteristics in ultrasound images, such as internal and external echoes, aspect ratio, border margins, and the presence or absence of cystic changes. However, ultrasound images are easily affected by speckle noise and echo disturbances, making it difficult to distinguish nodule boundaries. Furthermore, the diagnosis of benign or malignant thyroid nodules relies heavily on the professional knowledge and clinical experience of the sonographer, and the diagnostic results are significantly influenced by the physician's subjectivity. Therefore, a computer-aided thyroid ultrasound image nodule segmentation and classification system has practical application value.

[0088] To address at least one of the aforementioned technical problems, this application provides a method for constructing an auxiliary diagnostic model for thyroid nodules based on multimodal data. The method includes: acquiring multimodal thyroid nodule medical data; wherein the multimodal thyroid nodule medical data includes clinical data and image data; classifying the thyroid nodule medical data of each modality using a unimodal classification model to obtain intermediate classification results for each thyroid nodule corresponding to the medical data of each modality; and weightedly fusing all intermediate classification results to obtain a benign / malignant classification result for thyroid nodules based on the multimodal thyroid nodule medical data. This application's embodiment uses a unimodal classification model to classify the medical data of each modality and fuses the outputs of each unimodal classification model, eliminating the step of uniformly converting multimodal data into structured data. Therefore, it is more efficient and has higher scenario adaptability during data processing. In addition, the embodiments of this application can be used for the diagnosis, monitoring and treatment of Bethesda III thyroid nodules, assisting clinicians in improving the diagnostic efficiency of thyroid nodules, reducing the clinical burden, and providing important theoretical and technical support for the early identification and individualized treatment of benign and malignant Bethesda III thyroid nodules.

[0089] In addition, this application provides a thyroid nodule auxiliary diagnostic system based on multimodal data. The system includes: an acquisition module for acquiring multimodal thyroid nodule medical data of at least one patient; the multimodal thyroid nodule medical data includes clinical data and image data; an intermediate classification module for classifying the thyroid nodule medical data of each patient in each modality using a single-modal classification model to obtain intermediate classification results for each thyroid nodule corresponding to the thyroid nodule medical data of each patient in each modality; and a weighted fusion module for weighted fusion of all intermediate classification results of thyroid nodules of each patient to obtain the benign or malignant classification result of the thyroid nodule corresponding to each patient. The thyroid nodule auxiliary diagnostic system based on multimodal data in this application embodiment can process multimodal thyroid nodule medical data from multiple patients simultaneously. When classifying the multimodal thyroid nodule medical data, it classifies the thyroid nodule medical data of each patient for each modality, and then fuses the intermediate classification results to assist clinicians in improving the diagnostic efficiency of thyroid nodules, reducing the clinical burden, and providing important theoretical and technical support for the early identification and treatment of benign and malignant Bethesda III thyroid nodules.

[0090] Figure 1 This diagram illustrates a schematic flowchart of a method for constructing an auxiliary diagnostic model for thyroid nodules based on multimodal data according to an embodiment of this application; as shown. Figure 1 As shown, the method 100 for constructing a thyroid nodule auxiliary diagnostic model based on multimodal data according to an embodiment of this application may include the following steps S101, S102 and S103:

[0091] In step S101, multimodal thyroid nodule medical data is acquired.

[0092] The multimodal thyroid nodule medical data may include unstructured data such as clinical data and image data, or structured data such as text data.

[0093] Structured data, as defined here, refers to data with a fixed format that can be represented using a uniform structure (such as a table). Its characteristics include clearly defined relationships between data items, making it easy for computers to read and process directly. Unstructured data, on the other hand, lacks a fixed format and is difficult to represent using a uniform structure. Its content is complex and irregular, typically requiring specific technologies (such as natural language processing and image recognition) to be understood by computers. Because this application's embodiments employ a single-modal model to process each type of data separately, it exhibits strong compatibility with various data types and can handle multiple data types.

[0094] In step S102, the unimodal classification model is used to classify the thyroid nodule medical data of each modality to obtain the intermediate classification results of each thyroid nodule corresponding to the thyroid nodule medical data of each modality.

[0095] Among them, the classification model corresponding to the modality of the multimodal thyroid nodule medical data belongs to the unimodal model. A unimodal model refers to an artificial intelligence model that only processes one type of data (such as plain text, image, audio, etc.). In the data processing stage, it does not involve the fusion of multi-source data, but only merges the processing results of each unimodal model, eliminating the step of uniformly converting multimodal data into structured data. Therefore, it is more efficient and has a higher scene adaptability in the data processing stage.

[0096] In one embodiment of this application, the multimodal thyroid nodule medical data includes image data; the image data includes ultrasound images and cytological images. Figure 2 The diagram shown is a schematic flowchart of the process 200 for training a first classification model based on image data according to an embodiment of this application. The process 200 for training a first classification model based on image data according to an embodiment of this application may include steps S201, S202, S203, S204, and S205:

[0097] In step S201, data augmentation is performed on the image data samples to obtain multiple intermediate image data samples;

[0098] In step S202, at least two different image classification networks are constructed as the candidate first classification model. The collected medical image samples and their corresponding classification labels are used as the input of the candidate first classification model. During the training process, based on the transfer learning strategy, the pre-trained image feature extraction network is used as the basic model to extract features from the input image, obtain feature vectors, and then input them into the constructed fully connected layer classifier for feature processing and classification.

[0099] In step S203, the at least two candidate first classification models are validated using the five-fold cross-validation method to obtain the accuracy data of each candidate first classification model;

[0100] In step S204, based on the accuracy data of each candidate first classification model, the model with the best area under the curve (AUC) performance is selected from the at least two candidate first classification models as the first classification model, so that the first classification model can classify thyroid nodules as benign or malignant based on the image data.

[0101] In this embodiment, multiple machine learning algorithms are used to train on multiple image data samples to obtain at least two candidate first classification models. Then, the model with the best AUC performance is selected as the first classification model from these two candidate models. By using multiple machine learning algorithms to train the candidate first classification models, the model with the strongest fit to the task data distribution is selected from the multiple candidate first classification models, thereby improving the classification accuracy.

[0102] Furthermore, ultrasound images are greatly affected by equipment model, operator technique, and individual patient differences (such as fat thickness and organ location), resulting in often unstable data distribution. Different candidate primary classification models have varying tolerances for data noise and distribution shifts. By screening multiple candidate primary classification models, a more stable model can be found in diverse data, avoiding a sharp drop in performance due to data fluctuations.

[0103] In one example, such as Figure 3 As shown, step S201 involves performing data augmentation on the image data samples to obtain multiple intermediate image data samples, including steps S301, S302, and S303:

[0104] In step S301, a cropping operation is performed on each image data sample to convert each image data sample into an image of a preset size.

[0105] In step S302, the image of the preset size is standardized, wherein the standardization operation is based on the standard values ​​of the pre-trained model;

[0106] In step S303, the standardized image data samples are subjected to a preset number of random rotation operations to obtain the multiple intermediate image data samples at different angles.

[0107] Because actual acquired images (such as lesion images in medical imaging or object images in natural scenes) may have various rotation angles (e.g., the patient's body tilt during the photo shoot causing different lesion angles, or the camera angle being random during the photo shoot), or they may have a fixed degree of rotation (e.g., all images of the patient in an upright posture), models trained using these images may memorize irrelevant features such as "the image must be upright" (rather than the key features of the lesion itself). When testing or using the model, if it encounters images with other rotation angles, the model will make a misjudgment. Therefore, in this embodiment, the standardized image data samples are randomly rotated a preset number of times to generate more "new image data samples," that is, to simulate the "diversity" of images in real scenes, so as to avoid the model "memorizing" irrelevant features. In this way, after randomly rotating the preprocessed image data, the angle of the image changes, but the core features of the lesion (such as shape, texture, and edges) remain unchanged. After the model has seen enough samples with rotation angles during training, it will gradually ignore the irrelevant factor of "angle" and focus on the core features that are stable at any angle. As a result, it can still accurately identify images when faced with new and unseen images at any angle, making the trained model more robust.

[0108] In one example, such as Figure 4 As shown, step S203 uses five-fold cross-validation to validate the at least two candidate first classification models and obtains the accuracy data of each candidate first classification model, including steps S401, 402, 403, and 404:

[0109] In step S401, for any candidate first classification model, the plurality of intermediate image data samples are divided into five mutually exclusive subsets;

[0110] In step 402, one subset is used as the validation set and the other four subsets are used as the training set. The Adam optimization algorithm and cross-entropy loss function are used to perform five training runs and five validation runs to obtain five performance scores.

[0111] In step 403, the performance of each candidate first classification model is evaluated based on the area under the corresponding receiver operating characteristic curve. The average of the five performance scores is calculated as the performance evaluation data of the candidate first classification model.

[0112] In step 404, the best-performing model among the candidate first classification models is selected as the first classification model, so that the first classification model can classify thyroid nodules as benign or malignant based on the image data.

[0113] In this embodiment of the application, when using the five-fold cross-validation method to validate the at least two candidate first classification models, multiple intermediate image data samples are first divided into five "folds". Each time, multiple intermediate image data samples from four folds are randomly selected as the training set (for model training), and multiple intermediate image data samples from the other fold are selected as the validation set (for evaluating model performance).

[0114] For example: if there are 1000 samples of intermediate image data, then each fold contains 200 samples, and the 5 folds are denoted as Fold1, Fold2, Fold3, Fold4, and Fold5 respectively;

[0115] First iteration: Use Fold2-5 as the training set and Fold1 as the validation set;

[0116] Second loop: Use Fold1 and 3-5 as the training set, and Fold2 as the validation set;

[0117] The third iteration: Use Fold1-2 and 4-5 as the training set, and Fold3 as the validation set;

[0118] Fourth iteration: Use Fold1-3 and 5 as the training set and Fold4 as the validation set;

[0119] 5th iteration: Use Fold1-4 as the training set and Fold5 as the validation set.

[0120] Obviously, training with the above data samples will yield five intermediate machine learning models. In subsequent steps, these five intermediate machine learning models need to be optimized and selected.

[0121] In this embodiment, the Adam (Adaptive Moment Estimation) optimization algorithm is a commonly used optimization algorithm in deep learning. It is mainly used to adjust the parameters of a neural network model to minimize the loss function, thereby improving the training efficiency and performance of the model. The Adam optimization algorithm assumes that during model training, Adam updates the model parameters in each iteration by calculating the first moment (momentum term), calculating the second moment (adaptive term), correcting bias, and updating parameters. Adam combines momentum mechanism and adaptive learning rate, and has the advantages of fast convergence speed, high stability, and strong adaptability. It also performs well in various deep learning tasks such as image recognition, natural language processing, and recommendation systems.

[0122] Furthermore, the cross-entropy loss function is a commonly used loss function in machine learning, primarily for classification tasks (such as binary and multi-class classification), measuring the "degree of difference" between the model's predicted results and the true labels. Its main function is to quantify the model's prediction error by calculating the "cross-entropy" between the predicted probability distribution and the true probability distribution, thereby guiding the optimization of model parameters. Compared to other loss functions (such as mean squared error, MSE), cross-entropy loss directly calculates the loss based on the difference in probability distributions in classification tasks, which is more in line with the mathematical logic of the task. Moreover, the gradient of the cross-entropy loss is directly related to the prediction error, resulting in higher training efficiency. The cross-entropy loss function provides an efficient and reasonable loss metric for classification tasks by measuring the difference between the model's predicted probability distribution and the true probability distribution.

[0123] In this embodiment of the application, based on the verification results of five sets of intermediate machine learning models, the model parameters of each intermediate machine learning model are optimized, and then the robustness of each intermediate machine learning model is verified again. The intermediate machine learning model with the highest accuracy among the five sets of intermediate machine learning models is selected as any candidate first classification model, and the model parameters of the intermediate machine learning model with the highest accuracy are saved.

[0124] This application employs a five-fold cross-validation method, which optimizes the model parameters of each candidate first-class classification model and reduces the risk of overfitting. Traditional techniques use a single training-validation cycle to train machine learning models, which may lead to fluctuations in evaluation results due to the randomness of data splitting. The five-fold cross-validation method in this application weakens this randomness by averaging multiple different splitting combinations, resulting in a result closer to the model's true generalization ability. Furthermore, by covering all possible validation sets, the evaluation results are more robust.

[0125] In addition, by analyzing the fluctuation of the five validation results (such as the standard deviation), it is possible to determine whether there are problems such as uneven distribution or too many extreme samples in the data: if the validation result of a certain time is significantly lower than the others, the corresponding data subset may have a special distribution (such as class imbalance), indicating that further data processing is needed.

[0126] In one example, such as Figure 5 As shown, step S204 involves selecting the model with the best area under the curve performance from at least two candidate first classification models based on the accuracy data of each candidate first classification model. This first classification model then classifies thyroid nodules as benign or malignant based on the image data. This includes steps 501, 502, and 503.

[0127] In step 501, based on the accuracy data of each candidate first classification model, generate the Receiver Operating Characteristic (ROC) curves of each candidate first classification model;

[0128] In step 502, calculate the area under the corresponding ROC curve to evaluate the performance of each candidate first classification model;

[0129] In step 503, select the model with the best performance among each candidate first classification model as the first classification model, and enable the first classification model to classify the benign and malignant of thyroid nodules based on the image data.

[0130] Generally, the "curve" in the Area Under the Curve (AUC) refers to the Receiver Operating Characteristic (ROC) curve. The AUC, that is, the area between the ROC curve and the horizontal axis, reflects the classification performance of the model. The closer the value of the AUC is to 1, the stronger the discrimination ability of the model. For example,

[0131] AUC = 1: Perfect model, which can distinguish positive samples and negative samples 100% (the predicted probabilities of all positive samples are higher than those of negative samples);

[0132] 0.5 < AUC < 1: The model has a certain discrimination ability, and the closer the value is to 1, the stronger the discrimination ability;

[0133] AUC = 0.5: The model makes completely random guesses and has no actual discrimination ability;

[0134] AUC < 0.5: The performance of the model is worse than random guessing (usually can be corrected to > 0.5 by reversing the prediction results).

[0135] AUC is an index that quantifies the ability of a classification model to distinguish positive and negative samples through the area under the ROC curve. It does not depend on the classification threshold and can comprehensively evaluate the overall performance of the model, especially suitable for tasks with data imbalance and high requirements for discrimination ability (such as medical image classification). The closer the value of the Area Under the Curve (AUC) is to 1, the stronger the discrimination ability of the model and the higher the application reliability.

[0136] Simply put, the higher the AUC, the more accurately the model can distinguish positive samples and negative samples. For example, in ultrasonic image classification, a model with AUC = 0.9 can more reliably distinguish "malignant nodules" (positive samples) and "benign nodules" (negative samples) than a model with AUC = 0.7.

[0137] The ROC curve is a curve with "False Positive Rate (FPR)" on the horizontal axis and "True Positive Rate (TPR)" on the vertical axis. The horizontal axis of the ROC curve represents FPR (False Positive Rate), and the vertical axis represents TPR (True Positive Rate). The True Positive Rate (TPR) is the proportion of samples correctly predicted as positive by the model (i.e., the proportion of patients correctly diagnosed, reflecting the model's sensitivity and recall); the False Positive Rate (FPR) is the proportion of negative samples incorrectly predicted as positive by the model (i.e., the proportion of healthy individuals misdiagnosed as having the disease, reflecting the model's misdiagnosis rate). True Positive Rate (TPR): The proportion of samples that are actually positive that are correctly predicted as positive by the model (i.e., "hit rate"); False Positive Rate (FPR): The proportion of samples that are actually negative that are incorrectly predicted as positive by the model (i.e., "false alarm rate").

[0138] The formulas for calculating the horizontal axis FPR and the vertical axis TPR are as follows:

[0139] (1)

[0140] (2)

[0141] (3)

[0142] Wherein, TP represents true positives; FP represents false positives; TN represents true negatives; FN represents false negatives; FPR represents false positive rate; TPR represents true positive rate; and ACC represents accuracy.

[0143] It is worth noting that the image data includes ultrasound images or cytological images. Since both are images, they can both be trained using the methods described in steps S201 to S205. However, due to the differences between the two types of images, the machine learning algorithms used during model training may differ, resulting in different machine learning models. Specifically, the model trained using ultrasound images is a first ultrasound classification model, and the model trained using cytological images is a first cell classification model. The first ultrasound classification model is used to classify ultrasound images to obtain a first intermediate classification result; the first cell classification model is used to classify cytological images to obtain a second intermediate classification result.

[0144] The ultrasound image may include at least one of the following: ultrasound elastography, grayscale ultrasound image and / or color Doppler ultrasound image.

[0145] The applicant collected data from 487 cases who underwent surgery at the Affiliated Hospital of Southwest Medical University between January 2019 and March 2024, including general clinical data, ultrasound images, cytological images, and inflammatory markers. This data was then used to train an AI model. In one example, nine models trained using algorithms such as Convolutional Neural Networks (CNN) and Transformer, based on ultrasound images, are shown in Table 1.

[0146] Table 1: Nine models trained based on ultrasound images

[0147]

[0148] ResNet50 (Residual Network with 50 layers) is a classic deep convolutional neural network (CNN) model. It employs residual connections, allowing inputs to be directly superimposed on the output, enabling the network to more easily learn "residual mappings" and thus support deeper network structures. Furthermore, the 50-layer depth, while maintaining feature extraction capabilities, controls computational cost through residual blocks and bottleneck structures, making it easier to deploy.

[0149] In the DenseNet121 (Dense Convolutional Network) model, 121 represents the number of network layers. The DenseNet121 model employs dense connections, where each convolutional layer is directly connected to all previous layers, maximizing feature reuse, reducing parameter redundancy, and mitigating the vanishing gradient problem. It is highly parameter-efficient and suitable for tasks with small sample sizes (such as medical images), but it requires significant computation.

[0150] GoogleNet (Inception v1) employs the Inception module, which extracts multi-scale features in the same layer by using convolutional kernels of different sizes (1×1, 3×3, 5×5) and pooling layers in parallel, balancing the receptive field size and improving feature extraction efficiency. It significantly reduces parameters and computational cost while maintaining accuracy, and has performed exceptionally well in the ImageNet competition.

[0151] Vision Transformer (ViT) is a vision model based on the Transformer architecture. It segments images into multiple "pictures," processing them like text sequences. It captures global feature relationships through self-attention, freeing it from the reliance of CNNs on local receptive fields. It outperforms traditional CNNs on large datasets and is suitable for tasks requiring long-range feature associations (such as panoramic image analysis), but requires more data for training.

[0152] CNN (Convolutional Neural Network) is the basic framework of most of the above models (ResNet, DenseNet, etc.). It extracts local features (such as edges and textures) through convolution operations, compresses features by combining pooling layers, and automatically learns the hierarchical representation of images (from low-level features to high-level semantics).

[0153] Swin Transformer is an improved visual model based on Transformer. It divides the image into non-overlapping windows, calculates self-attention within the window (reducing computation), and achieves cross-window feature interaction through window sliding, taking into account both local details and global relationships.

[0154] VGG19 uses a stack of only 3×3 convolutional kernels and 2×2 pooling layers to improve performance by increasing the number of layers. The structure is simple but has a huge number of parameters (approximately 140 million parameters).

[0155] InceptionV3 is an improved version of GoogleNet. It features a lightweight design for the Inception module (e.g., replacing n×n convolutions with 1×n and n×1 convolutions) and introduces Batch Normalization to accelerate training, reduce parameters and computational cost, and improve accuracy. It is suitable for scenarios with limited computing resources and is often used as a pre-trained model for transfer learning.

[0156] The above examples can be analyzed using the Delong test (P-value), which compares the ResNet50 model with each other model to obtain a P-value. If the P-value is less than a preset value (e.g., less than 0.05), then the data in Table 1 is statistically significant. Given that the data in Table 1 is statistically significant, it can be seen from Table 1 that the ResNet50 model has the highest accuracy (ACC) at 0.6634. Therefore, the ResNet50 model can be selected as the primary ultrasound classification model.

[0157] In another example, nine models trained using algorithms such as Convolutional Neural Networks (CNN) and Transformer, based on cytological images, are shown in Table 2:

[0158] Table 2: Various models trained based on cytological images

[0159]

[0160] Among them, ResNet101 is a deep residual network based on convolutional neural networks (CNN). It solves the gradient vanishing problem during deep network training by using "residual connections" (skip connections). It can stably train a deep network with 101 layers and has stronger feature extraction capabilities, but it requires more computation. It is suitable for tasks that require deep features, such as image classification and medical image analysis (e.g., tumor detection).

[0161] DenseNet121 is a densely connected network based on CNN. It adopts a "dense connection" mechanism, where each convolutional layer is directly connected to all previous layers, maximizing feature reuse and achieving high parameter efficiency. It is suitable for small sample scenarios (such as scarce medical image data), reducing parameter redundancy while maintaining accuracy, and is sensitive to detailed features.

[0162] GoogleNet (Inception v1) is a multi-scale feature extraction network based on CNN. It captures multi-scale features in the same layer by using convolutional kernels of different sizes (1×1, 3×3, 5×5) and pooling layers in parallel, balancing computational cost and accuracy.

[0163] Vision Transformer (ViT) is a vision model based on the Transformer architecture. It segments images into "image patches" and captures global feature relationships through a "self-attention mechanism" just like processing text sequences, making it suitable for large datasets.

[0164] CNN (Convolutional Neural Network) is a basic framework for convolutional neural networks. It extracts local features (such as edges and textures) through "convolution operations", compresses features by combining pooling layers, and automatically learns a hierarchical representation of images (from low-level features to high-level semantics).

[0165] ResNet50 is a residual network based on CNN. It has a 50-layer network structure and solves the performance degradation problem of deep networks through residual connections, achieving a balance between accuracy and computational cost. It can be applied to image classification, object detection (such as as the backbone network of Faster R-CNN), and pre-trained models for transfer learning.

[0166] VGG19 is a deep network based on CNN. It uses pure 3×3 convolutional kernels and stacked 2×2 pooling layers. It improves feature extraction capabilities by increasing the number of layers (19 layers). It has a simple structure but a large number of parameters.

[0167] InceptionV3 is an improved model in the Inception series based on CNN. It is a lightweight optimization of the original Inception module (such as replacing n×n convolution with 1×n and n×1 convolutions), and introduces Batch Normalization to accelerate training, reducing the amount of computation while improving accuracy.

[0168] Swin Transformer is an improved visual model based on Transformer. It balances global features and computational efficiency through "window self-attention" and "sliding window mechanism", and performs well in dense prediction tasks such as object detection and semantic segmentation.

[0169] The above examples can be analyzed using the Delong test (P-value). This involves comparing the Swing-transformer model with each other model to obtain a P-value. If the P-value is less than a preset value (e.g., less than 0.05), the data in Table 2 is considered statistically significant. Given that the data in Table 2 are statistically significant, it can be seen from Table 2 that the Swing-transformer model has the highest accuracy (ACC) at 0.7059. Therefore, the Swing-transformer model can be chosen as the primary cell classification model.

[0170] In another embodiment of this application, the medical data is clinical data. For example... Figure 6 The diagram shown is a schematic flowchart of the process 600 for training a second classification model based on clinical data according to an embodiment of this application. The process 600 for training a second classification model based on clinical data according to an embodiment of this application may include steps S601, S602, S603, S604, and S605:

[0171] In step S601, multiple clinical data samples are acquired;

[0172] In step S602, data preprocessing is performed on the multiple clinical data samples; wherein, the data preprocessing includes: quantizing and encoding the categorical attributes, deleting features with missing values ​​exceeding a preset proportion, and performing max-min normalization on the numerical features to improve the stability of model training;

[0173] In step S603, based on multiple clinical data after data preprocessing, each sample is represented as a feature vector and input into at least two different machine learning algorithms for modeling training to obtain at least two candidate second classification models;

[0174] In step S604, the candidate second classification model is trained and validated using the five-fold cross-validation method. During the training process, the model parameters are iteratively updated using the stochastic gradient descent optimization algorithm, and the grid search method is used for tuning.

[0175] In step S605, the model with the best area under the curve performance from the at least two candidate second classification models is selected as the second classification model, so that the second classification model classifies thyroid nodules as benign or malignant based on the clinical data, and obtains a third intermediate classification result.

[0176] The clinical data include at least one of the following: gender, age, height, weight, past medical history, duration of thyroid nodule discovery, growth rate of thyroid nodule, and Bethesda classification of thyroid nodule.

[0177] Clinical data can be obtained from electronic medical record (EMR) systems. Clinical data can include both structured and unstructured data. For example, information such as patient symptoms recorded by doctors in EMRs is structured data, while image data from patient medical histories is unstructured data.

[0178] In one example, such as Figure 7 As shown, step S605, which selects the model with the best area under the curve performance from the at least two candidate second classification models, includes steps S701 and S702:

[0179] In step S701, a receiver operating characteristic curve is generated based on the probability values ​​of the classification results of each candidate second classification model, and the corresponding area under the curve is calculated to evaluate the performance of each candidate second classification model.

[0180] In step S702, the candidate second classification model with the best area under the curve performance is selected as the second classification model.

[0181] When training the secondary classification model using multiple clinical data samples, algorithms such as linear regression, SVM (Support Vector Machine), and decision tree can be used for training. The training results are shown in Table 3.

[0182] Table 3: Five models trained based on clinical data

[0183]

[0184] Logistic Regression is a linear classification model that uses the sigmoid function to map the output (continuous value) of linear regression to probability values ​​between 0 and 1 to determine the probability of a sample belonging to a certain category. The model is simple, trains quickly, and is highly interpretable (the coefficients directly reflect the influence weight of features on the results).

[0185] SVC (Support Vector Classification) is a non-linear classification model based on kernel functions. It determines the optimal classification hyperplane by finding "support vectors" (samples closest to the classification boundary) to maximize the margin between samples of different classes. It maps low-dimensional non-linear data to a high-dimensional space through kernel functions (such as RBF or linear kernel) to achieve linear separability. It is suitable for small samples and high-dimensional data (such as text classification) and has strong generalization ability.

[0186] KNN (K-Nearest Neighbors) is an instance-based lazy learning model (non-parametric model). It does not have an explicit training process; during prediction, it calculates the distance (such as Euclidean distance) between the sample to be predicted and the "K nearest samples" in the training set, determining the result based on the majority class (classification) or the average (regression) of these K samples. Its model is simple, easy to understand, requires no training, and can predict directly.

[0187] Random Forest is an ensemble learning model based on decision trees. It constructs multiple decision trees by randomly sampling (samples and features), and the final result is determined by voting (classification) or averaging (regression) among the multiple trees, reducing the risk of overfitting from a single decision tree.

[0188] XGBoost (Extreme Gradient Boosting) is an ensemble learning model (boosting strategy) based on decision trees. It iteratively builds multiple "weak decision trees," each focusing on correcting the prediction errors of the preceding trees (by optimizing the loss function through gradient descent), and finally weights and combines the results of all trees. It boasts extremely high accuracy, performs exceptionally well in various competitions (such as Kaggle), and is widely used for structured data tasks.

[0189] The above examples can be analyzed using the Delong test (P-value). This involves comparing the Logistic regression model with each other to obtain a P-value. If the P-value is less than a preset value (e.g., less than 0.05), then the data in Table 3 is statistically significant. Given that the data in Table 3 is statistically significant, it can be seen from Table 3 that the Logistic regression model has the highest accuracy (ACC) at 0.7000. Therefore, the Logistic regression model can be chosen as the secondary classification model.

[0190] In step S103, all intermediate classification results of thyroid nodules are weighted and fused to obtain the benign and malignant classification results of thyroid nodules based on multimodal thyroid nodule medical data.

[0191] In one embodiment of this application, such as Figure 8 As shown, step S103 involves weighted fusion of all intermediate classification results for thyroid nodules to obtain benign / malignant classification results for thyroid nodules based on multimodal thyroid nodule medical data, including steps S801, S802, S803, S804, and S805:

[0192] In step S801, the first intermediate classification result output by the first ultrasound classification model is received;

[0193] In step S802, the second intermediate classification result output by the first cell classification model is received;

[0194] In step S803, the third intermediate classification result output by the second classification model is received;

[0195] In step S804, any combination of the first intermediate classification result, the second intermediate classification result, and the third intermediate classification result is weighted and fused to obtain multiple sets of intermediate benign and malignant classification results;

[0196] In step S805, the result with the best area under the curve (AUC) performance is selected from the multiple intermediate benign and malignant classification results as the benign and malignant classification result of the thyroid nodule.

[0197] Among them, for the results predicted by the first ultrasound classification model, the predicted disease probability is:

[0198] (4)

[0199] (5)

[0200] in, Represents a set of n ultrasound images; This represents the nth ultrasound image; This represents the classification result of the first ultrasound classification model for n ultrasound images; This represents the classification result of the first ultrasound classification model for the nth ultrasound image.

[0201] After averaging the predictions for all ultrasound images (all ultrasound images for each patient for diagnostic purposes), the prediction output of the first ultrasound classification model (prediction for each patient for diagnostic purposes) is as follows:

[0202] (6)

[0203] in, This represents the first predicted probability of benign or malignant transformation (i.e., the first intermediate classification result); This represents the classification result of the first ultrasound classification model for n ultrasound images; Indicates the number of ultrasound images; i and k represent integers.

[0204] Specifically, for cytological images in the test set predicted by the optimal cytological image classification model, the predicted disease probability is:

[0205] (7)

[0206] (8)

[0207] in, This indicates that the first cell classification model is targeting... Classification results of Zhang's ultrasound images; This indicates that the first cell classification model is for the first cell. Classification results of Zhang's ultrasound images; express A collection of cytological images; This represents the nth cytological image classified by the first cell classification model.

[0208] After averaging the image results for each patient, the second benign / malignant prediction probability value predicted by the first cell classification model constructed based on the cytological images is:

[0209] (9)

[0210] in, This represents the second benign / malignant prediction probability value (i.e., the second intermediate classification result) predicted by the first cell classification model. This represents the classification result of the first cell classification model for i cytological images; The number of cytological images is represented; i and k represent integers.

[0211] For the clinical data of n patients in the test set, the probability predicted by the second classification model is:

[0212] (10)

[0213] (11)

[0214] in, This indicates that the second classification model is for the first... Predicted probability values ​​for 100 patients; This represents the set of probability values ​​predicted by the second classification model (i.e., the third intermediate classification result). This represents the set of clinical data for n patients; This represents the clinical data of the nth patient.

[0215] It is worth noting that the first ultrasound classification model and the first cell classification model, when processing the corresponding image data, also process the data for each individual patient. For example, they can distinguish the data of different patients by assigning patient numbers, and the final output is the prediction result for each patient. Similarly, the second classification model also processes the clinical data corresponding to each patient.

[0216] The predicted probability values ​​of the first ultrasound classification model, the first cell classification model, and the second classification model are fused together, and the best-performing fusion combination is selected to obtain the final benign and malignant classification results of thyroid nodules based on multimodal data (ultrasound images, cytological images, and clinical data).

[0217] The fusion result of ultrasound image and cytology image diagnostic prediction is expressed as follows:

[0218] (12)

[0219] in, The weight values ​​represent the probability predictions of ultrasound images; The weight values ​​represent the probability predictions for cytological images; This represents the probability value predicted by the first ultrasound classification model; This represents the probability value predicted by the first cell classification model.

[0220] The fusion result of ultrasound images and clinical feature diagnostic predictions is represented as follows:

[0221] (13)

[0222] in, The weight values ​​represent the probability predictions of ultrasound images; The weight values ​​represent the probability predictions of clinical data; This represents the probability value predicted by the first ultrasound classification model; This represents the probability value predicted by the second classification model.

[0223] The fusion result of cytological images and clinical feature diagnostic predictions is represented as follows:

[0224] (14)

[0225] in, The weight values ​​represent the probability predictions for cytological images; The weight values ​​represent the probability predictions of clinical data; This represents the probability value predicted by the first cell classification model; This represents the probability value predicted by the second classification model.

[0226] The weighted fusion result of the diagnostic results from ultrasound images, cytological images, and clinical data is as follows:

[0227] (15)

[0228] in, The weight values ​​represent the probability predictions of ultrasound images; The weight values ​​represent the probability predictions for cytological images; The weight values ​​represent the probability predictions of clinical data; This represents the probability value predicted by the second classification model; This represents the probability value predicted by the first cell classification model; This represents the probability value predicted by the first ultrasound classification model.

[0229] In one example, the weight of the first benign / malignant prediction probability can be used as the area under the curve performance value of the first ultrasound classification model; the weight of the second benign / malignant prediction probability can be used as the area under the curve performance value of the first cell classification model; and the weight of the third benign / malignant prediction probability can be used as the area under the curve performance value of the second classification model.

[0230] For example, combining Tables 1, 2 and 3, the first weight is 0.7419, the second weight is 0.7789, and the third weight is 0.7050.

[0231] In other examples, the importance of various data can be determined based on historical image data, historical clinical data, etc., and the values ​​of the first weight, the second weight, and the third weight can be assigned according to the importance of the various data. For example, the first weight is 0.5, the second weight is 0.3, and the third weight is 0.2.

[0232] This application employs a unimodal classification model to classify medical data for each modality, and fuses the outputs of each unimodal classification model. This eliminates the step of uniformly converting multimodal data into structured data, resulting in higher efficiency and scenario adaptability during data processing. Furthermore, this application can diagnose, monitor, and treat Bethesda III thyroid nodules, assisting clinicians in improving diagnostic efficiency and reducing clinical burden. It provides important theoretical and technical support for the early identification and individualized treatment of benign and malignant Bethesda III thyroid nodules.

[0233] like Figure 9 The diagram shown is a schematic flowchart of a method for constructing an auxiliary diagnostic model for thyroid nodules based on multimodal data, according to another embodiment of this application. According to the method for constructing an auxiliary diagnostic model for thyroid nodules according to this embodiment, firstly, ultrasound images, cytological images, and clinical data are acquired. Then, a single-modality first ultrasound classification model is trained based on the ultrasound images, and a first intermediate classification result is output from the first ultrasound classification model; a single-modality first cell classification model is trained based on the cytological images, and a second intermediate classification result is output from the first cell classification model; a single-modality second classification model is trained based on the clinical data, and a third intermediate classification result is output from the second classification model. Then, the first intermediate classification result, the second intermediate classification result, and the third intermediate classification result are weighted and fused to obtain the benign / malignant classification result of the thyroid nodules based on multimodal data.

[0234] The embodiments of this application do not require the design of complex multimodal data fusion modules (e.g., feature alignment between text and images, intermodal attention mechanisms, etc.), resulting in a clearer model architecture that is easier to understand and implement. Moreover, using a single-modal model during the data processing stage leads to faster inference speed and lower response latency, thus improving the overall model processing speed.

[0235] like Figure 10 The diagram shown is a schematic flowchart of a thyroid nodule auxiliary diagnosis method based on multimodal data according to an embodiment of this application. The thyroid nodule auxiliary diagnosis method 1000 based on multimodal data according to an embodiment of this application may include the following steps S1001 and S1002;

[0236] In step S1001, multimodal thyroid nodule medical data of at least one patient is acquired; the multimodal thyroid nodule medical data includes clinical data and / or image data;

[0237] In step S1002, the thyroid nodule medical data of each patient in each modality is classified using a single-modality classification model to obtain intermediate classification results of each thyroid nodule corresponding to the thyroid nodule medical data of each patient in each modality.

[0238] In step S1003, the intermediate classification results of all thyroid nodules for each patient are weighted and fused to obtain the benign or malignant classification result of the thyroid nodules for each patient.

[0239] The thyroid nodule auxiliary diagnosis method based on multimodal data in this application embodiment can simultaneously process the clinical data and / or image data of thyroid nodules from multiple patients when using a thyroid nodule auxiliary diagnosis model. When classifying the data, it processes the multimodal thyroid nodule medical data corresponding to each patient, and diagnoses, monitors, and treats Bethesda III thyroid nodules for each patient. This assists clinicians in improving the diagnostic efficiency of thyroid nodules, reducing the clinical burden, and provides important theoretical and technical support for the early identification and treatment of benign and malignant Bethesda III thyroid nodules.

[0240] The following is combined with Figure 11 The thyroid nodule auxiliary diagnostic system 1100 based on multimodal data according to an embodiment of this application will be described. Figure 11 A schematic block diagram of a thyroid nodule auxiliary diagnostic system 1100 based on multimodal data according to an embodiment of this application is shown. The thyroid nodule auxiliary diagnostic system 1100 based on multimodal data according to an embodiment of this application includes an acquisition module 1101, an intermediate classification module 1102, and a weighted fusion module 1103.

[0241] The acquisition module 1101 is used to acquire multimodal thyroid nodule medical data of at least one patient; the multimodal thyroid nodule medical data includes clinical data and image data.

[0242] The intermediate classification module 1102 is used to classify the thyroid nodule medical data of each patient in each modality using a single-modality classification model, and obtain the intermediate classification results of each thyroid nodule corresponding to the thyroid nodule medical data of each patient in each modality.

[0243] The weighted fusion module 1103 is used to weight and fuse the intermediate classification results of all thyroid nodules for each patient to obtain the benign or malignant classification result of the thyroid nodules for each patient.

[0244] The image data includes ultrasound images and cytological images;

[0245] In one embodiment of this application, such as Figure 12As shown, the intermediate classification module 1102 includes a first classification submodule 1201, a second classification submodule 1202, and a third classification submodule 1203; wherein,

[0246] The first classification submodule 1201 is used to classify the ultrasound image using a first ultrasound classification model to obtain a first intermediate classification result;

[0247] The second classification submodule 1202 is used to classify the cytology image using the first cell classification model to obtain a second intermediate classification result;

[0248] The third classification submodule 1203 is used to classify the clinical data using the second classification model to obtain a third intermediate classification result.

[0249] The first ultrasound classification model is constructed based on the following algorithms: 50-layer residual network algorithm, DenseNet121 algorithm, GoogleNet algorithm, Vision Transformer algorithm, Vision Transformer algorithm, convolutional neural network algorithm, Swing Transformer algorithm, VGG19 algorithm, or InceptionV3 algorithm.

[0250] The first cell classification model is constructed based on the following algorithms: 50-layer residual network algorithm, DenseNet121 algorithm, GoogleNet algorithm, Vision Transformer algorithm, convolutional neural network algorithm, 50-layer residual network algorithm, VGG19 algorithm, InceptionV3 algorithm, or Swing Transformer algorithm.

[0251] The second classification model is built based on the following algorithms: logistic regression, support vector classification, K-nearest neighbors, random forest, or extreme gradient boosting.

[0252] In one example, the verification submodule 1202 is specifically used for:

[0253] The weighted fusion module is specifically used to calculate the first sum of the products of the first intermediate classification result and the first weight, the second intermediate classification result and the second weight, and the third intermediate classification result and the third weight; and to calculate the second sum of the first weight, the second weight, and the third weight, and divide the first sum by the second sum to obtain the benign or malignant classification result of the thyroid nodules for each patient. Please refer to formula (15) above.

[0254] In one example, the first weight includes the area under the curve (AUC) performance value of the first ultrasound classification model; the second weight includes the AUC performance value of the first cell classification model; and the third weight includes the AUC performance value of the second classification model. For example, referring to Tables 1, 2, and 3, the first weight is 0.7419, the second weight is 0.7789, and the third weight is 0.7050.

[0255] In other examples, the importance of various data can be determined based on historical image data, historical clinical data, etc., and the first weight, the second weight, and the third weight can be assigned according to the importance of the various data. For example, the first weight is 0.5, the second weight is 0.3, and the third weight is 0.2.

[0256] The ultrasound images in this embodiment may include ultrasound elastography, grayscale ultrasound images, and / or color Doppler ultrasound images; the clinical data include at least one of the following: gender, age, height, weight, past medical history, duration of thyroid nodule discovery, growth rate of thyroid nodules, and Bethesda classification of thyroid nodules.

[0257] Additionally, embodiments of this application may include at least one acquisition device (not shown in the figures), wherein the acquisition device acquires at least two types of medical data of the thyroid nodules and sends the at least two types of medical data to the acquisition module 1101, the intermediate classification module 1102, and the weighted fusion module 1103 for processing. The acquisition device includes at least one of the following: an ultrasound diagnostic system, a pathology information management system (PIMS), and an electronic medical record (EMR).

[0258] The thyroid nodule auxiliary diagnostic system based on multimodal data in this application embodiment can process multimodal thyroid nodule medical data from multiple patients simultaneously. When classifying the multimodal thyroid nodule medical data, it classifies the thyroid nodule medical data of each patient for each modality, and then fuses the intermediate classification results to assist clinicians in improving the diagnostic efficiency of thyroid nodules, reducing the clinical burden, and providing important theoretical and technical support for the early identification and treatment of benign and malignant Bethesda III thyroid nodules.

[0259] The following is combined with Figure 13 The present application describes a thyroid nodule auxiliary diagnostic terminal 1300 based on multimodal data, wherein, Figure 13A schematic block diagram of a thyroid nodule auxiliary diagnostic terminal 1300 based on multimodal data according to an embodiment of this application is shown.

[0260] The thyroid nodule auxiliary diagnostic terminal 1300 based on multimodal data includes:

[0261] The memory 1301 and the processor 1302 store a computer program that is executed by the processor. When the computer program is executed by the processor, it causes the processor to perform the thyroid nodule auxiliary diagnosis method as described above.

[0262] like Figure 13 As shown, the thyroid nodule auxiliary diagnosis terminal 1300 based on multimodal data includes: one or more memories 1301 and one or more processors 1302. The memories 1301 store a computer program that is run by the processors 1302. When the computer program is run by the processors 1302, the processors 1302 execute the thyroid nodule auxiliary diagnosis method based on multimodal data described above.

[0263] The thyroid nodule auxiliary diagnosis terminal 1300 based on multimodal data can be part or all of a computer device that can implement a thyroid nodule auxiliary diagnosis method based on multimodal data through software, hardware, or a combination of software and hardware.

[0264] like Figure 13 As shown, the thyroid nodule auxiliary diagnostic terminal 1300 based on multimodal data includes one or more memories 1301, one or more processors 1302, a display (not shown), and a communication interface, etc., which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). It should be noted that... Figure 13 The components and structures of the thyroid nodule auxiliary diagnostic terminal 1300 shown are merely exemplary and not limiting. The thyroid nodule auxiliary diagnostic terminal 1300 may also have other components and structures as needed.

[0265] Memory 1301 is used to store various data and executable program instructions generated during the operation of related methods, such as storing various application programs or algorithms that implement various specific functions. It may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0266] The processor 1302 may be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other processing units with data processing and / or instruction execution capabilities, and may be other components in the thyroid nodule auxiliary diagnostic terminal 1300 to perform the desired functions.

[0267] In one example, the thyroid nodule auxiliary diagnostic terminal 1300 also includes an output device that can output various information (such as images or sounds) to the outside (e.g., a user), and may include one or more of a display device, a speaker, etc.

[0268] The communication interface can be any known communication protocol interface, such as a wired interface or a wireless interface. The communication interface may include one or more serial ports, USB interfaces, Ethernet ports, WiFi, wired networks, DVI interfaces, device integrated interconnect modules, or other suitable ports, interfaces, or connections.

[0269] Furthermore, according to embodiments of this application, a storage medium is also provided, on which program instructions are stored. When executed by a computer or processor, these program instructions are used to perform corresponding steps of the thyroid nodule auxiliary diagnostic method of this application. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0270] Furthermore, according to embodiments of this application, a computer program product is also provided, wherein the computer program / instructions, when executed by a processor, implement the thyroid nodule auxiliary diagnosis method as described above.

[0271] The thyroid nodule auxiliary diagnosis system, thyroid nodule auxiliary diagnosis terminal, computer program product, and storage medium based on multimodal data of the present application embodiments have the same advantages as the aforementioned thyroid nodule auxiliary diagnosis method because they can realize the aforementioned thyroid nodule auxiliary diagnosis method based on multimodal data.

[0272] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0273] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0274] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0275] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0276] Similarly, it should be understood that, in order to streamline this application and aid in understanding one or more of the various inventive aspects, features of this application may sometimes be grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with features fewer than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0277] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or elements of any method or apparatus so disclosed may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0278] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0279] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0280] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0281] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A method for constructing an auxiliary diagnostic model for thyroid nodules based on multimodal data, characterized in that, The method includes: Acquire multimodal thyroid nodule medical data; wherein, the multimodal thyroid nodule medical data includes clinical data and image data; The medical data of thyroid nodules in each modality were classified using a single-modality classification model to obtain the intermediate classification results of each thyroid nodule corresponding to the medical data of each modality. All intermediate classification results of thyroid nodules were weighted and fused to obtain the benign and malignant classification results of thyroid nodules based on multimodal thyroid nodule medical data; Where the medical data for thyroid nodules includes ultrasound images and cytological images, the method further includes: performing data augmentation on the image data samples to obtain multiple intermediate image data samples; constructing at least two different image classification networks as candidate first classification models, using the collected medical image samples and their corresponding classification labels as inputs to the candidate first classification models; during training, based on a transfer learning strategy, using a pre-trained image feature extraction network as a base model to extract features from the input images, obtaining feature vectors, and then inputting them into a constructed fully connected layer classifier for feature processing and classification; using a five-fold cross-validation method to validate the at least two candidate first classification models and obtain accuracy data for each candidate first classification model; based on the accuracy data of each candidate first classification model, selecting the model with the best area under the curve performance from the at least two candidate first classification models as the first classification model, so that the first classification model can classify thyroid nodules as benign or malignant based on the image data; Wherein, when the image data includes ultrasound images or cytological images, the first classification model includes a first ultrasound classification model or a first cell classification model; wherein, the first ultrasound classification model is used to classify the ultrasound images to obtain a first intermediate classification result; the first cell classification model is used to classify the cytological images to obtain a second intermediate classification result; Where the thyroid nodule medical data includes clinical data, the method further includes: acquiring multiple clinical data samples; performing data preprocessing on the multiple clinical data samples; representing each sample as a feature vector based on the preprocessed clinical data, and inputting it into at least two different machine learning algorithms for modeling training to obtain at least two candidate second classification models; training and validating the candidate second classification models using a five-fold cross-validation method, iteratively updating the model parameters using a stochastic gradient descent optimization algorithm during training, and fine-tuning using a grid search method; selecting the model with the best area under the curve performance from the at least two candidate second classification models as the second classification model, so that the second classification model classifies thyroid nodules into benign or malignant based on the clinical data to obtain a third intermediate classification result; wherein, the data preprocessing includes: quantizing and encoding classification attributes, deleting features with missing values ​​exceeding a preset proportion, and performing max-min normalization on numerical features to improve the stability of model training; The process involves weighted fusion of all intermediate classification results for thyroid nodules to obtain a benign / malignant classification result for thyroid nodules based on multimodal thyroid nodule medical data. This includes: receiving a first intermediate classification result from a first ultrasound classification model; receiving a second intermediate classification result from a first cell classification model; receiving a third intermediate classification result from a second classification model; weighted fusion of any combination of the first, second, and third intermediate classification results to obtain multiple sets of intermediate benign / malignant classification results; and selecting the result with the best area under the curve from these multiple sets of intermediate benign / malignant classification results as the benign / malignant classification result for the thyroid nodule.

2. The method according to claim 1, characterized in that, Data augmentation operations are performed on image data samples to obtain multiple intermediate image data samples, including: Each image data sample is cropped to convert it into an image of a preset size. The image of the preset size is standardized, wherein the standardization operation is based on the standard values ​​of the pre-trained model; The standardized image data samples are subjected to a preset number of random rotation operations to obtain multiple intermediate image data samples at different angles.

3. The method according to claim 1, characterized in that, Five-fold cross-validation was used to validate at least two candidate primary classification models, and the accuracy data of each candidate primary classification model was obtained, including: For any candidate first classification model, the plurality of intermediate image data samples are divided into five mutually exclusive subsets; One subset was used as the validation set, and the other four subsets were used as the training set. Five training iterations and five validation iterations were performed using the Adam optimization algorithm and the cross-entropy loss function to obtain five performance scores. The performance of each candidate first classification model is evaluated based on the area under the corresponding receiver operating characteristic curve. The average of the five performance scores is calculated as the performance evaluation data of the candidate first classification model. The best-performing model among the candidate first classification models is selected as the first classification model, which then classifies thyroid nodules as benign or malignant based on the image data.

4. The method according to claim 1, characterized in that, Selecting the model with the best area under the curve performance from the at least two candidate second classification models as the second classification model includes: Receiver operating characteristic curves are generated based on the probability values ​​of the classification results of each candidate secondary classification model, and the corresponding area under the curve is calculated to evaluate the performance of each candidate secondary classification model. The candidate second classification model with the best area under the curve performance was selected as the second classification model.

5. The method according to claim 1, characterized in that, The first weight includes the area under the curve performance value of the first ultrasound classification model; The second weight includes the area under the curve performance value of the first cell classification model; The third weight includes the area under the curve performance value of the second classification model.

6. The method according to any one of claims 1 to 5, characterized in that, The ultrasound images include at least one of the following: ultrasound elastography, grayscale ultrasound images, and color Doppler ultrasound images; The clinical data include at least one of the following: gender, age, height, weight, past medical history, duration of thyroid nodule discovery, growth rate of thyroid nodule, and Bethesda classification of thyroid nodule.

7. A thyroid nodule auxiliary diagnostic system based on multimodal data, characterized in that, The system includes: The acquisition module is used to acquire multimodal thyroid nodule medical data from at least one patient; the multimodal thyroid nodule medical data includes clinical data and image data; The intermediate classification module is used to classify the thyroid nodule medical data of each patient in each modality using a single-modality classification model, and obtain the intermediate classification results of each thyroid nodule corresponding to the thyroid nodule medical data of each patient in each modality. The weighted fusion module is used to weight and fuse the intermediate classification results of all thyroid nodules for each patient to obtain the benign or malignant classification result of the thyroid nodules for each patient. Specifically, the intermediate classification module is used to perform data augmentation on image data samples to obtain multiple intermediate image data samples when the thyroid nodule medical data includes ultrasound images and cytological images; construct at least two different image classification networks as candidate first classification models, using the collected medical image samples and their corresponding classification labels as input to the candidate first classification models; during training, based on a transfer learning strategy, using a pre-trained image feature extraction network as the base model to extract features from the input images, obtaining feature vectors, which are then input into a constructed fully connected layer classifier for feature processing and classification; verifying the at least two candidate first classification models using a five-fold cross-validation method to obtain accuracy data for each candidate first classification model; and selecting the model with the best area under the curve performance from the at least two candidate first classification models as the first classification model, enabling the first classification model to classify thyroid nodules as benign or malignant based on the image data; wherein the image data includes ultrasound images and cytological images; and the first classification model includes a first ultrasound classification model or a first cell classification model; The intermediate classification module includes a first classification submodule and a second classification submodule. The first classification submodule is used to classify the ultrasound image using a first ultrasound classification model to obtain a first intermediate classification result. The second classification submodule is used to classify the cytology image using a first cell classification model to obtain a second intermediate classification result. The intermediate classification module further includes a third classification submodule. This third classification submodule is used to: acquire multiple clinical data samples when the thyroid nodule medical data includes clinical data; perform data preprocessing on the multiple clinical data samples; represent each sample as a feature vector based on the preprocessed clinical data and input it into at least two different machine learning algorithms for modeling training to obtain at least two candidate second classification models; train and validate the candidate second classification models using a five-fold cross-validation method, iteratively update the model parameters using a stochastic gradient descent optimization algorithm during training, and fine-tune the model using a grid search method; select the model with the best area under the curve performance from the at least two candidate second classification models as the second classification model, enabling the second classification model to classify thyroid nodules as benign or malignant based on the clinical data, thus obtaining a third intermediate classification result; wherein the data preprocessing includes: quantizing and encoding classification attributes, deleting features with missing values ​​exceeding a preset proportion, and performing max-min normalization on numerical features to improve the stability of model training; Specifically, the weighted fusion module is used to calculate the first sum of the product of the first intermediate classification result and the first weight, the product of the second intermediate classification result and the second weight, and the product of the third intermediate classification result and the third weight, and to calculate the second sum of the first weight, the second weight, and the third weight, and to divide the first sum by the second sum to obtain the benign or malignant classification result of the thyroid nodules for each patient.

8. The system according to claim 7, characterized in that, The first ultrasound classification model is constructed based on the following algorithms: 50-layer residual network algorithm, DenseNet121 algorithm, GoogleNet algorithm, Vision Transformer algorithm, Vision Transformer algorithm, convolutional neural network algorithm, Swing Transformer algorithm, VGG19 algorithm, or InceptionV3 algorithm. The first cell classification model is constructed based on the following algorithms: 50-layer residual network algorithm, DenseNet121 algorithm, GoogleNet algorithm, Vision Transformer algorithm, convolutional neural network algorithm, 50-layer residual network algorithm, VGG19 algorithm, InceptionV3 algorithm, or Swing Transformer algorithm. The second classification model is built based on the following algorithms: logistic regression, support vector classification, K-nearest neighbors, random forest, or extreme gradient boosting.

9. The system according to claim 7, characterized in that, The first weight includes the area under the curve performance value of the first ultrasound classification model; The second weight includes the area under the curve performance value of the first cell classification model; The third weight includes the area under the curve performance value of the second classification model.

10. The system according to any one of claims 7 to 9, characterized in that, The ultrasound images include at least one of the following: ultrasound elastography, grayscale ultrasound images, and / or color Doppler ultrasound images; The clinical data include at least one of the following: gender, age, height, weight, past medical history, duration of thyroid nodule discovery, growth rate of thyroid nodule, and Bethesda classification of thyroid nodule.

11. A thyroid nodule auxiliary diagnostic terminal based on multimodal data, characterized in that, The terminal includes: The system includes a memory and a processor, wherein the memory stores a computer program that is executed by the processor, and the computer program, when executed by the processor, causes the processor to perform the method for constructing a thyroid nodule auxiliary diagnostic model based on multimodal data as described in any one of claims 1-6.

12. A storage medium, characterized in that, The storage medium includes a computer program stored on it, which, when run by a processor, causes the processor to execute the method for constructing a thyroid nodule auxiliary diagnostic model based on multimodal data as described in any one of claims 1-6.

13. A computer program product, characterized in that, When the computer program / instructions are executed by the processor, they implement the method for constructing a thyroid nodule auxiliary diagnostic model based on multimodal data as described in any one of claims 1-6.

Images