Thyroid blood flow analysis method based on multi-modal data
By using multimodal data encoder and attention mechanism fusion characteristics in thyroid blood flow analysis, and using LoRA fine-tuning Chinese large model for analysis, the problem of manual labeling in the existing technology is solved, and the accurate analysis and diagnostic accuracy of thyroid blood flow is achieved.
Patent Information
- Application Number
- CN202411943544.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-05-02
AI Technical Summary
The existing thyroid blood flow analysis methods require manual labeling, which consumes a lot of time and energy, and is prone to image color deviations and wrong blood flow extraction, which affects the classification accuracy and cannot fully utilize the correlation between multimodal data.
The thyroid blood flow analysis method based on multimodal data was adopted, and features were extracted using the improved ResNet-50 image feature encoder, Sentence-Bert text data encoder and 1D convolutional detection data encoder were used to extract features, feature fusion was performed through attention mechanism and tensor fusion, and blood flow analysis was performed using LoRA fine-tuned Chinese big model Baichuan-7B.
It realizes accurate quantification and analysis of thyroid blood flow, reduces the work burden of doctors, improves analysis efficiency and diagnostic accuracy, and has significant clinical application value.
Smart Images
Figure CN119908754A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a thyroid blood flow analysis method, belonging to the technical field of medical image processing. Background Art
[0002] In the diagnosis of thyroid diseases, ultrasound imaging technology has become an important tool for doctors to evaluate the thyroid gland and its nodules due to its non-invasiveness and high cost-effectiveness. Doctors usually combine ultrasound images, Doppler images, the patient's pathological texts and test data to comprehensively analyze the health of the thyroid gland, and then determine whether the patient has thyroid diseases such as hyperthyroidism and Hashimoto's thyroiditis, and make a judgment on whether the nodules are benign or malignant. At present, the incidence of thyroid diseases is increasing year by year, and accurate analysis based on multimodal data plays a vital role in the early diagnosis and treatment of the disease.
[0003] Traditional thyroid blood flow analysis methods usually rely on doctors to manually mark the thyroid area or nodule area in the Doppler image, combined with image processing technology (such as corrosion, color space conversion, etc.), or use specific software to extract blood flow information, analyze the blood flow velocity, and the blood flow area of the surrounding and internal areas, and then combine the test data and use traditional classification algorithms (such as support vector machines, multi-layer perceptrons, etc.) for classification.
[0004] This process not only requires a lot of time and effort due to the need for manual annotation, but also causes image color deviation and incorrect blood flow extraction problems, further affecting the accuracy of classification. In addition, traditional methods often fail to fully utilize the correlation between multimodal data and cannot fully integrate information such as images, pathology, and test data, resulting in limited accuracy and clinical application value of diagnostic recommendations, and the operation is complex and inefficient, making it difficult to meet the growing clinical needs.
[0005] In response to these challenges, the present invention proposes a thyroid blood flow analysis method based on multimodal data. This method automatically performs deep encoding and feature fusion on multimodal data, and inputs it into the fine-tuned Chinese large model Baichuan-7B for analysis, thereby achieving accurate quantification and analysis of thyroid blood flow conditions, including detailed descriptions of the blood flow position, area, and velocity of the thyroid gland and nodules, while providing doctors with further diagnostic advice. This method can effectively integrate multimodal information, automatically perform blood flow analysis and diagnosis, reduce the workload of doctors, improve analysis efficiency and diagnostic accuracy, and has significant clinical application value. Summary of the invention
[0006] In order to solve the problems that the existing thyroid blood flow analysis method not only requires a lot of time and effort due to manual labeling, but also causes image color deviation and erroneous blood flow extraction, the present invention proposes a thyroid blood flow analysis method based on multimodal data.
[0007] The technical solution adopted by the present invention to solve the above-mentioned problem is: the steps of the present invention include:
[0008] Step 1: For the input data; thyroid ultrasound images, Doppler images and clinical text data, use an image feature encoder based on an improved ResNet-50, a text data encoder based on Sentence-Bert, and a detection data encoder composed of 1D convolution, batch normalization and 1D average pooling to extract the multimodal features required for thyroid blood flow analysis;
[0009] Step 2: Convert the features of different modalities into a unified format, and then use a multimodal feature fusion method based on attention mechanism and tensor fusion to fuse the multimodal features, and use a regularized loss function to reduce the redundancy between modal features;
[0010] Step 3: Perform image processing on the data set to obtain accurate and quantitative data set annotations;
[0011] Step 4: Based on the Chinese large model Baichuan-7B, the LoRA fine-tuning method is adopted, and an overall semantic loss is added to the loss function. The quantized blood flow analysis data set is used for fine-tuning to obtain a professional thyroid blood flow analysis large model; the multimodal data is input into the thyroid blood flow analysis model to obtain a detailed description of the location, area, and speed of the blood flow of the thyroid gland and nodules, and suggestions for further diagnosis are given.
[0012] Furthermore, step 1 specifically includes:
[0013] The image feature encoder used for thyroid ultrasound images and Doppler images replaces the 3*3 convolution of the convolutional layer in ResNet-50 with a spatial attention unit, that is, the output x∈(b,c,h,w) of the previous convolutional block is transformed by using 1*1 convolution to reduce the number of channels and then obtain T a ∈(b,h*w,c / 8), and then use 1*1 convolution to reduce the number of channels and transform the shape to get T k ∈(b,c / 8,h*w), after batch matrix multiplication, softmax and shape transformation are used to obtain the attention feature T att ∈(b,h*w,h*w), and T obtained by shape transformation after 1*1 convolution v∈(b,c,h*w) is multiplied and transformed into T′∈(b,c,h,w), and the residual connection is made with the input to get the output T of the convolution, where b represents the batch size, c represents the number of channels, and h and w represent the height and width of the feature map respectively; in addition, the activation function ReLU in the convolution layer is replaced by GELU;
[0014] The text data encoder used for unstructured text data in clinical text data uses Sentence-Bert to embed the text data into a vector representation through a pre-trained model, and then uses uniform manifold approximation and projection technology to reduce the dimension of the features;
[0015] The test data encoder used for structured test data in clinical text data includes 1D convolution, batch normalization and 1D average pooling. After 5 stackings, the test data is subjected to feature extraction and dimensionality enhancement.
[0016] Furthermore, in addition to the patient's pathological text, the unstructured text data also converts the information obtained from the previous task: thyroid nodule segmentation based on thyroid ultrasound images and thyroid Doppler images into descriptive text, which is encoded together with other text data and input into the large model for automatic thyroid blood flow analysis.
[0017] Furthermore, step 2 specifically includes:
[0018] Process the various modes obtained by the encoder and convert their shapes into a unified format to obtain v i ,i∈{1,2,3,4}; Then a multimodal feature fusion method based on attention mechanism and tensor fusion is adopted, that is, v is firstly connected by a fully connected layer. i becomes Where l represents the length and n represents the number of cases; then, through bilinear transformation, v is calculated i and all other modes, denoted as V i The attention weights between them are also of length l; these attention weights are applied to the feature v i ′, we get the enhanced feature v after information interaction i Specifically, the calculation formula is as follows:
[0019]
[0020] In formula (1), Next, the feature fusion method of tensor fusion is used to aggregate the features of all modalities together to form a fusion feature v that includes single-modal features, bi-modal features, and multi-modal features. The formula is as follows:
[0021]
[0022] In formula (2), Where M is the number of modes; finally, it is flattened and passed through two fully connected layers to obtain the fused feature v p In addition, a regularization loss function is added. This function constrains the redundant expression of the model in the fusion process by calculating the difference between the sum of the nuclear norms of each modal feature and the nuclear norm of the modal feature set. The formula is as follows:
[0023]
[0024] In formula (3), ||·|| represents the calculation of the nuclear norm of the matrix.
[0025] Furthermore, step 3 specifically includes:
[0026] In addition to the doctor's description of the distribution and morphological information of the blood flow of the thyroid gland and nodules, the annotation of the data set also quantifies the blood flow, including the quantification of blood flow area and flow velocity information. The specific operation method is as follows:
[0027] Convert the Doppler image to HSV format, extract the blood flow part by setting the threshold of brightness and saturation, and find the blood flow of the corresponding part according to the mask obtained by the previous thyroid and nodule segmentation task;
[0028] The distribution of blood flow around and inside the thyroid gland and nodules is quantified, and the blood flow speed is quantified according to color to obtain the quantified data set annotation.
[0029] Furthermore, step 4 specifically includes:
[0030] The pre-trained Chinese large model Baichuan-7B was fine-tuned using the LoRA method to optimize the q and v modules of the attention mechanism in the transformer structure to obtain a professional thyroid blood flow analysis large model; in addition, an overall semantic loss was added to the loss function:
[0031] The cosine similarity is calculated by embedding the actual blood flow description and the predicted blood flow description, and then summed after RELU, logarithmic operation and negation operation to obtain the loss of overall semantic similarity. The formula is as follows:
[0032]
[0033] In formula (4), f RELU (·) and CS(·) represent RELU and cosine similarity calculations, respectively. Label and pred represent the embedding vectors of the dataset annotation and prediction results, respectively. Finally, the multimodal data are input into the trained thyroid blood flow analysis model to obtain the blood flow analysis results of the thyroid and nodules, and suggestions for further diagnosis are given.
[0034] The beneficial effects of the present invention are as follows: the present invention is different from previous methods in that the comprehensive analysis capability is weak, the blood flow analysis is inaccurate, the operation is inconvenient, and the like; the present invention targets at a variety of medical data of patients, uses different feature encoders for encoding, obtains accurate feature representation results, and then uses a feature fusion method based on attention mechanism, tensor fusion, and regularization loss to reduce feature redundancy and enhance the representation of each modal feature; then, the Chinese large model Baichuan-7B is fine-tuned using LoRA to obtain accurate quantification and analysis of thyroid blood flow conditions, and at the same time provide doctors with further diagnostic suggestions; the present invention can effectively integrate multimodal information, automatically perform blood flow analysis and diagnosis, can reduce the workload of doctors, improve analysis efficiency and diagnostic accuracy, has significant clinical application value, and provides more intelligent and convenient technical support for early screening and diagnosis of thyroid diseases. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION
[0036] Specific implementation method 1: Figure 1 As shown, a thyroid blood flow analysis method based on multimodal data includes the following specific steps:
[0037] Step 1: For the input data; thyroid ultrasound images, Doppler images and clinical text data, use an image feature encoder based on an improved ResNet-50, a text data encoder based on Sentence-Bert, and a detection data encoder composed of 1D convolution, batch normalization and 1D average pooling to extract the multimodal features required for thyroid blood flow analysis;
[0038] The image feature encoder used for thyroid ultrasound images and Doppler images replaces the 3*3 convolution of the convolutional layer in ResNet-50 with a spatial attention unit, that is, the output x∈(b,c,h,w) of the previous convolutional block is transformed by using 1*1 convolution to reduce the number of channels and then obtain T a ∈(b,h*w,c / 8), and then use 1*1 convolution to reduce the number of channels and transform the shape to get T k ∈(b,c / 8,h*w), after batch matrix multiplication, softmax and shape transformation are used to obtain the attention feature T att ∈(b,h*w,h*w), and T obtained by shape transformation after 1*1 convolution v∈(b,c,h*w) is multiplied and transformed into T′∈(b,c,h,w), and the residual connection is made with the input to get the output T of the convolution, where b represents the batch size, c represents the number of channels, and h and w represent the height and width of the feature map respectively; in addition, the activation function ReLU in the convolution layer is replaced by GELU;
[0039] The text data encoder used for unstructured text data in clinical text data uses Sentence-Bert to embed the text data into a vector representation through a pre-trained model, and then uses uniform manifold approximation and projection technology to reduce the dimension of the features;
[0040] The test data encoder used for structured test data in clinical text data includes 1D convolution, batch normalization, and 1D average pooling. After five stackings, the test data is subjected to feature extraction and dimension enhancement.
[0041] In addition to the patient's pathological text, the unstructured text data also converts the information obtained from the previous task: thyroid nodule segmentation based on thyroid ultrasound images and thyroid Doppler images into descriptive text, which is encoded together with other text data and input into the thyroid blood flow automatic analysis model;
[0042] Step 2: Convert the features of different modalities into a unified format, and then use a multimodal feature fusion method based on attention mechanism and tensor fusion to fuse the multimodal features, and use a regularized loss function to reduce the redundancy between modal features;
[0043] Process the various modes obtained by the encoder and convert their shapes into a unified format to obtain v i ,i∈{1,2,3,4}; Then a multimodal feature fusion method based on attention mechanism and tensor fusion is adopted, that is, v is firstly connected by a fully connected layer. i becomes Where l represents the length and n represents the number of cases; then, through bilinear transformation, v is calculated i and all other modes, denoted as V i The attention weights between them are also of length l; these attention weights are applied to the feature v i ′, we get the enhanced feature v after information interaction i Specifically, the calculation formula is as follows:
[0044]
[0045] In formula (1), Next, the feature fusion method of tensor fusion is used to aggregate the features of all modalities together to form a fusion feature v that includes single-modal features, bi-modal features, and multi-modal features. The formula is as follows:
[0046]
[0047] In formula (2), Where M is the number of modes; finally, it is flattened and passed through two fully connected layers to obtain the fused feature v p In addition, a regularization loss function is added. This function constrains the redundant expression of the model in the fusion process by calculating the difference between the sum of the nuclear norms of each modal feature and the nuclear norm of the modal feature set. The formula is as follows:
[0048]
[0049] In formula (3), ||·|| represents the calculation of the nuclear norm of the matrix;
[0050] Step 3: Perform image processing on the data set to obtain accurate and quantitative data set annotations;
[0051] In addition to the doctor's description of the distribution and morphological information of the blood flow of the thyroid gland and nodules, the annotation of the data set also quantifies the blood flow, including the quantification of blood flow area and flow velocity information. The specific operation method is as follows:
[0052] Convert the Doppler image to HSV format, extract the blood flow part by setting the threshold of brightness and saturation, and find the blood flow of the corresponding part according to the mask obtained by the previous thyroid and nodule segmentation task;
[0053] Quantify the distribution of blood flow around and inside the thyroid gland and nodules, and quantify the blood flow speed based on color to obtain the quantified data set annotation;
[0054] Step 4: Based on the Chinese large model Baichuan-7B, the LoRA fine-tuning method is adopted, and an overall semantic loss is added to the loss function. The quantized blood flow analysis data set is used for fine-tuning to obtain a professional thyroid blood flow analysis large model; the multimodal data is input into the thyroid blood flow analysis model to obtain a detailed description of the location, area, and speed of the blood flow of the thyroid gland and nodules, and further diagnostic suggestions are given;
[0055] The pre-trained Chinese large model Baichuan-7B was fine-tuned using the LoRA method to optimize the q and v modules of the attention mechanism in the transformer structure to obtain a professional thyroid blood flow analysis large model; in addition, an overall semantic loss was added to the loss function:
[0056] The cosine similarity is calculated by embedding the actual blood flow description and the predicted blood flow description, and then summed after RELU, logarithmic operation and negation operation to obtain the loss of overall semantic similarity. The formula is as follows:
[0057]
[0058] In formula (4), f RELU (·) and CS(·) represent RELU and cosine similarity calculations, respectively. Label and pred represent the embedding vectors of the dataset annotation and prediction results, respectively. Finally, the multimodal data are input into the trained thyroid blood flow analysis model to obtain the blood flow analysis results of the thyroid and nodules, and suggestions for further diagnosis are given.
[0059] Example
[0060] The present invention provides a thyroid blood flow analysis method based on multimodal data. First, the patient's thyroid ultrasound image, Doppler image, and clinical text data are obtained, and then encoded using respective encoders; then, the features of different modalities are converted into a unified format, and the multimodal features are fused using a multimodal feature fusion method to obtain a rich feature representation and reduce the redundancy between modal features; then, the data set is image processed to obtain an accurate and quantitative data set annotation; then the large model Baichuan-7B is fine-tuned to obtain a professional thyroid blood flow analysis large model; finally, the multimodal data is input into the thyroid blood flow analysis model to obtain a detailed description of the location, area, speed, etc. of the blood flow of the thyroid gland and nodules, and suggestions for further diagnosis are given. Figure 1 As shown, the specific steps are as follows:
[0061] Data preprocessing: The data set is strictly screened, desensitized, and enhanced. First, samples with information loss and abnormalities are screened out, and patient personal information is desensitized to ensure data privacy security. Then, through a series of data enhancement operations, such as image flipping, rotation, contrast change, etc., the diversity and robustness of the data are further expanded. Finally, the data is reasonably split into training sets and test sets to ensure the effectiveness and fairness of model training and evaluation.
[0062] In step 1, the patient's thyroid ultrasound image, Doppler image, and clinical text data are extracted using an image feature encoder based on an improved ResNet-50, a text data encoder based on Sentence-Bert (S-Bert), and a detection data encoder composed of 1D convolution, batch normalization, and 1D average pooling to extract the multimodal features required for thyroid blood flow analysis.
[0063] The image feature encoder used for thyroid ultrasound images and Doppler images replaces the 3*3 convolution of the convolutional layer in ResNet-50 with a spatial attention unit, that is, the output x∈(b, c, h, w) of the previous convolutional block is transformed by using 1*1 convolution to reduce the number of channels and then obtain T q ∈(b, h*w, c / 8), and then use 1*1 convolution to reduce the number of channels and transform the shape to get T k ∈(b, c / 8, h*w), after batch matrix multiplication, softmax and shape transformation are used to obtain the attention feature T att ∈(b, h*w, h*w), and T obtained by shape transformation after 1*1 convolution v ∈(b, c, h*w) is multiplied and transformed into T′∈(b, c, h, w), and the residual connection is made with the input to get the output T of the convolution, where b represents the batch size, c represents the number of channels, and h and w represent the height and width of the feature map respectively; in addition, the activation function ReLU in the convolution layer is replaced by GELU;
[0064] For the text data encoder used for unstructured text data in clinical text data, we use Sentence-Bert to embed the text data into vector representation through a pre-trained model, and then use the Uniform Manifold Approximation and Projection (UMAP) technology to reduce the feature dimension;
[0065] The test data encoder used for structured test data in clinical text data includes 1D convolution, batch normalization, and 1D average pooling. After five stackings, the test data is subjected to feature extraction and dimension enhancement.
[0066] Through the above operations, due to the use of customized encoders, we can fully convert the patient's pathological information into high-dimensional vectors, ensuring that the richness and completeness of the information are fully preserved, providing a more accurate and comprehensive basis for subsequent analysis.
[0067] Execute step 2 to convert the features of the above different modalities into a unified format, and then use a multimodal feature fusion method based on attention mechanism and tensor fusion to fuse the multimodal features, and use a regularized loss function to reduce the redundancy between modal features.
[0068] Among them, in addition to the patient's pathological text, the unstructured text data in the text data also converts the information obtained from the previous task: thyroid nodule segmentation task based on thyroid ultrasound images and thyroid Doppler images (location coordinates, morphology, size, etc. of the thyroid gland and nodules) into descriptive text, which is encoded together with other text data and input into the thyroid blood flow automatic analysis large model. By adding information such as the location, morphology, and size of the thyroid and nodules to the thyroid blood flow analysis model, the analysis area can be accurately located, and accurate blood flow analysis can be performed on the thyroid and nodules respectively. This optimization not only improves the accuracy of blood flow analysis, but also can more effectively identify lesion areas, providing stronger support for clinical diagnosis.
[0069] Next, the modal features obtained above are processed and their shapes are converted into a unified format to obtain v i , i∈{1, 2, 3, 4}; then a multimodal feature fusion method based on attention mechanism and tensor fusion is adopted, that is, v is firstly connected by a fully connected layer. i becomes Where l represents the length and n represents the number of cases; then, through bilinear transformation, v is calculated i Compared with all other modes (denoted as V i ), whose length is also l; apply these attention weights to feature v′ i We obtain the enhanced feature v″ after information interaction. i ; Specifically, the calculation formula is as follows:
[0070]
[0071] In the formula Next, the feature fusion method of tensor fusion is used to aggregate the features of all modalities together to form a fusion feature v that includes single-modal features, bi-modal features, and multi-modal features. The formula is as follows:
[0072]
[0073] In the formula Where M is the number of modes; finally, it is flattened and passed through two fully connected layers to obtain the fused feature v p In addition, a regularization loss function is added. This function constrains the redundant expression of the model in the fusion process by calculating the difference between the sum of the nuclear norms of each modal feature and the nuclear norm of the modal feature set. The formula is as follows:
[0074]
[0075] Here, ||·|| represents the calculation of the nuclear norm of the matrix.
[0076] Through the above operations, features of different modalities can better interact with each other and be effectively integrated together, while removing redundant information. The final fused features show stronger expressiveness and comprehensiveness. This optimization improves the model's ability to understand and analyze complex data, making it more accurate and efficient in practical applications.
[0077] Execute step three to perform image processing on the dataset to obtain accurate and quantitative dataset annotation.
[0078] In addition to the doctor's description of the distribution and morphology of the blood flow of the thyroid gland and nodules, the annotation of the data set used also quantifies the blood flow, including the quantification of blood flow area, flow velocity and other information. The specific operation method is: convert the Doppler image to HSV format, set the brightness and saturation thresholds to 50, extract the blood flow part, and according to the mask obtained by the previous thyroid and nodule segmentation task, reduce the diameter to 0.8 times the original through continuous corrosion to obtain the internal thyroid and nodule area (the boundary shape remains unchanged), and the rest is the surrounding area. Then quantify the distribution of blood flow around and inside the thyroid and nodules, such as the richness of thyroid blood flow and the richness of blood flow inside the nodule. In addition, according to the flow velocity scale in the picture, the blood flow speed is quantified according to the color of the blood flow, and finally the quantified data set annotation is obtained.
[0079] Through the above operations, the annotation of the data set is improved, so the final analysis model can obtain quantitative analysis results. These results provide clinicians with more precise and intuitive references, helping to make more accurate diagnosis and treatment decisions.
[0080] In step 4, the pre-trained Chinese large model Baichuan-7B is fine-tuned using the LoRA (Low-Rank Adaptation) method using the processed training set, and the q and v modules of the attention mechanism in the transformer structure are optimized to obtain a professional thyroid blood flow analysis large model; in addition, an overall semantic loss is added to the loss function: the cosine similarity is calculated by the embedding vectors of the actual blood flow description and the predicted blood flow description, and the sum is obtained after RELU, logarithmic operation and negation operation to obtain the overall semantic similarity loss, the formula is as follows:
[0081]
[0082] Among them, f RELU (·) and CS(·) represent RELU and cosine similarity calculations, respectively. Label and pred represent the embedding vectors of the dataset annotation and prediction results, respectively. Finally, the multimodal data of the test set is input into the trained thyroid blood flow analysis model to obtain the blood flow analysis results of the thyroid and nodules, and provide suggestions for further diagnosis.
[0083] Through the above operations, the fine-tuned large model has excellent multimodal data processing capabilities, and has deeply learned the professional knowledge of thyroid blood flow analysis, thus giving it accurate analysis capabilities. Through targeted training, the model can not only grasp key features such as blood flow velocity and vascular morphology, but also effectively identify lesions and abnormal patterns, and can output more accurate blood flow analysis results and diagnostic suggestions. Its description is more reasonable and highly consistent with the doctor's judgment, providing strong support for clinical diagnosis.
[0084] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.
Claims
1. A thyroid blood flow analysis method based on multimodal data, characterized in that: The specific steps include: Step 1: For the input data; thyroid ultrasound images, Doppler images and clinical text data, use an image feature encoder based on an improved ResNet-50, a text data encoder based on Sentence-Bert, and a detection data encoder composed of 1D convolution, batch normalization and 1D average pooling to extract the multimodal features required for thyroid blood flow analysis; Step 2: Convert the features of different modalities into a unified format, and then use a multimodal feature fusion method based on attention mechanism and tensor fusion to fuse the multimodal features, and use a regularized loss function to reduce the redundancy between modal features; Step 3: Perform image processing on the data set to obtain accurate and quantitative data set annotations; Step 4: Based on the Chinese large model Baichuan-7B, the LoRA fine-tuning method is adopted, and an overall semantic loss is added to the loss function. The quantized blood flow analysis data set is used for fine-tuning to obtain a professional thyroid blood flow analysis large model; the multimodal data is input into the thyroid blood flow analysis model to obtain a detailed description of the location, area, and speed of the blood flow of the thyroid gland and nodules, and suggestions for further diagnosis are given.
2. The thyroid blood flow analysis method based on multimodal data according to claim 1, characterized in that: Step 1 specifically includes: The image feature encoder used for thyroid ultrasound images and Doppler images replaces the 3*3 convolution of the convolutional layer in ResNet-50 with a spatial attention unit, that is, the output x∈(b,c,h,w) of the previous convolutional block is transformed by using 1*1 convolution to reduce the number of channels and then obtain T a ∈(b,h*w,c / 8), and then use 1*1 convolution to reduce the number of channels and transform the shape to get T k ∈(b,c / 8,h*w), after batch matrix multiplication, softmax and shape transformation are used to obtain the attention feature T att ∈(b,h*w,h*w), and T obtained by shape transformation after 1*1 convolution v ∈(b,c,h*w) is multiplied and transformed into T′∈(b,c,h,w), and the residual connection is made with the input to get the output T of the convolution, where b represents the batch size, c represents the number of channels, and h and w represent the height and width of the feature map respectively; in addition, the activation function ReLU in the convolution layer is replaced by GELU; The text data encoder used for unstructured text data in clinical text data uses Sentence-Bert to embed the text data into a vector representation through a pre-trained model, and then uses uniform manifold approximation and projection technology to reduce the dimension of the features; The test data encoder used for structured test data in clinical text data includes 1D convolution, batch normalization and 1D average pooling. After 5 stackings, the test data is subjected to feature extraction and dimensionality enhancement.
3. The thyroid blood flow analysis method based on multimodal data according to claim 2, characterized in that: In addition to the patient's pathological text, the unstructured text data also converts the information obtained from the previous task: thyroid nodule segmentation based on thyroid ultrasound images and thyroid Doppler images into descriptive text, which is encoded together with other text data and input into the large model for automatic thyroid blood flow analysis.
4. The thyroid blood flow analysis method based on multimodal data according to claim 1, characterized in that: Step 2 specifically includes: Process the various modes obtained by the encoder and convert their shapes into a unified format to obtain v i ,i∈{1,2,3,4}; Then a multimodal feature fusion method based on attention mechanism and tensor fusion is adopted, that is, v is firstly connected by a fully connected layer. i becomes Where l represents the length and n represents the number of cases; then v is calculated through bilinear transformation i and all other modes, denoted as V i The attention weights between them are also of length l; these attention weights are applied to the feature v i ′, we get the enhanced feature v after information interaction i Specifically, the calculation formula is as follows: In formula (1), Next, the feature fusion method of tensor fusion is used to aggregate the features of all modalities together to form a fusion feature v that includes single-modal features, bi-modal features, and multi-modal features. The formula is as follows: In formula (2), Where M is the number of modes; finally, it is flattened and passed through two fully connected layers to obtain the fused feature v p ; In addition, a regularization loss function is added. This function constrains the redundant expression of the model in the fusion process by calculating the difference between the sum of the nuclear norms of each modal feature and the nuclear norm of the modal feature set. The formula is as follows: In formula (3), ||·|| represents the calculation of the nuclear norm of the matrix.
5. The thyroid blood flow analysis method based on multimodal data according to claim 1, characterized in that: Step 3 specifically includes: In addition to the doctor's description of the distribution and morphological information of the blood flow of the thyroid gland and nodules, the annotation of the data set also quantifies the blood flow, including the quantification of blood flow area and flow velocity information. The specific operation method is as follows: Convert the Doppler image to HSV format, extract the blood flow part by setting the threshold of brightness and saturation, and find the blood flow of the corresponding part according to the mask obtained by the previous thyroid and nodule segmentation task; The distribution of blood flow around and inside the thyroid gland and nodules is quantified, and the blood flow speed is quantified according to color to obtain the quantified data set annotation.
6. The thyroid blood flow analysis method based on multimodal data according to claim 1, characterized in that: Step 4 specifically includes: The pre-trained Chinese large model Baichuan-7B was fine-tuned using the LoRA method to optimize the q and v modules of the attention mechanism in the transformer structure to obtain a professional thyroid blood flow analysis large model; in addition, an overall semantic loss was added to the loss function: The cosine similarity is calculated by embedding the actual blood flow description and the predicted blood flow description, and then summed after RELU, logarithmic operation and negation operation to obtain the loss of overall semantic similarity. The formula is as follows: In formula (4), f RELU (·) and CS(·) represent RELU and cosine similarity calculations, respectively. Label and pred represent the embedding vectors of the dataset annotation and prediction results, respectively. Finally, the multimodal data are input into the trained thyroid blood flow analysis model to obtain the blood flow analysis results of the thyroid and nodules, and suggestions for further diagnosis are given.