Benign breast disease diagnosis system based on multimodal data

Abstract

The present invention provides a benign breast disease diagnosis system, storage medium and electronic device based on multimodal data, which relate to the field of computer-aided diagnosis. In the present invention, the diagnostic information of images and texts is effectively integrated, and the fused comprehensive feature vector is used to perform accurate classification diagnosis of benign breast diseases, helping doctors make accurate clinical decisions. The proposed weighted fusion method not only improves the accuracy of diagnosis, but also ensures the efficiency and reliability of the entire diagnostic process, and is suitable for clinical diagnosis applications of benign breast diseases. The proposed two-stage large model processing flow combines the advantages of structured data generation and classification prediction, making the diagnosis of benign breast diseases more efficient and accurate. In addition, the combination of multiple deep learning technologies and attention mechanisms can significantly improve the accuracy and robustness of image classification, reduce the risk of misdiagnosis, and show excellent potential in actual clinical applications.

Description

Technical Field

[0001] The present invention relates to the field of computer-aided diagnosis, and in particular to a benign breast disease diagnosis system, storage medium and electronic equipment based on multimodal data. Background Art

[0002] Computer-aided diagnosis (CAD) is an interdisciplinary technology that uses computer vision and artificial intelligence to process medical images (typically radiological and pathological images) to identify diseases.

[0003] Among related technologies, computer-aided diagnosis systems have gradually been applied to the diagnosis of breast diseases. For example, through the automatic analysis of imaging data using technologies such as convolutional neural networks, significant progress has been made in the classification of benign and malignant tumors and the subclassification of malignant tumors.

[0004] However, research and application of deep learning technologies for benign breast diseases remain relatively limited. Existing deep learning applications in this field primarily focus on the identification and classification of malignant tumors, neglecting the refined diagnosis of benign lesions. Furthermore, current diagnostic systems primarily rely on image data for analysis. This single-modality analysis approach may not provide sufficient information support for the complex diagnosis of benign breast diseases, thus affecting the accuracy and comprehensiveness of diagnostic results. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the shortcomings of the existing technology, the present invention provides a benign breast disease diagnosis system, storage medium and electronic device based on multimodal data, which solves the technical problems of single technical means and insufficient integration of multimodal data.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A multimodal data-based benign breast disease diagnosis system, comprising:

[0010] A raw data acquisition module is used to acquire and pre-process diagnostic reports and ultrasound images of patients with benign breast diseases;

[0011] A text data diagnosis prediction module is used to obtain a first diagnostic classification result based on a preset two-stage large model for the preprocessed diagnosis report; wherein the two-stage large model adopts a fine-tuned LLaMA-3-8B model;

[0012] An image data diagnosis prediction module is used to obtain a second diagnostic classification result based on a preset multiple deep learning network for the preprocessed ultrasound image;

[0013] The diagnosis conclusion acquisition module is used to quantize the first diagnosis classification result and the second diagnosis classification result respectively, and obtain the final diagnosis classification result after weighted fusion.

[0014] Preferably, the method of obtaining a first diagnostic classification result based on a preset two-stage large model for the pre-processed diagnostic report includes:

[0015] In the first stage, the LoRA method is used to fine-tune the q_proj layer and v_proj layer of the first LLaMA-3-8B model, and the preprocessed diagnosis report is used as the input of the fine-tuned first LLaMA-3-8B model to structure the text data;

[0016] In the second stage, the LoRA method is used to fine-tune the q_proj layer and v_proj layer of the second LLaMA-3-8B model, and the structured text data is used as the input of the fine-tuned second LLaMA-3-8B model to obtain the first diagnostic classification result.

[0017] Preferably, the multiple deep learning networks include convolutional neural networks, multiple attention mechanisms, and Transformer modules; wherein the multiple attention mechanisms include SE modules, ECA modules, and CBAM modules in sequence.

[0018] Preferably, the convolutional neural network is ResNET34; the second diagnostic classification result is obtained based on a preset multiple deep learning network for the preprocessed ultrasound image; including:

[0019] The preprocessed ultrasound image is used as the input of the ResNET34 to extract preliminary features;

[0020] The preliminary features are used as the input of the multi-attention mechanism, and global features are enhanced by the SE module, local features are enhanced by the ECA module, and spatial features are enhanced by the CBAM module to obtain an enhanced feature map;

[0021] The enhanced feature map is used as the input of the Transformer module to model global features, and after global average pooling, the second diagnostic classification result is obtained through a fully connected layer.

[0022] Preferably, the step of quantizing the first diagnosis classification result and the second diagnosis classification result respectively and performing weighted fusion to obtain the final diagnosis classification result comprises:

[0023] Use the BERT model to vectorize the first diagnosis classification result and the second diagnosis classification result respectively to obtain a high-dimensional vector representation of the text and a high-dimensional vector representation of the image;

[0024] weightedly fusing the high-dimensional vector representation of the text and the high-dimensional vector representation of the image to obtain a fused vector;

[0025] The fusion vector is used as input and passes through a fully connected layer and a Softmax layer in sequence to obtain the final diagnosis classification result.

[0026] Preferably, the fusion vector is expressed as:

[0027] v fused =α·v text +β·v image

[0028] Among them, v fused is the fusion vector, v text is the high-dimensional vector representation of the text, v image is the high-dimensional vector representation of the image, α and β are the corresponding weight coefficients, and α>β.

[0029] Preferably, the pretreatment includes:

[0030] The diagnostic report is standardized, including text cleaning, removal of irrelevant information, formatting text structure and adjusting text length; and the ultrasound image is standardized, including image scaling and normalization.

[0031] A storage medium stores a computer program for diagnosing benign breast diseases based on multimodal data, wherein the computer program enables a computer to control the benign breast disease diagnosis system as described above to perform benign breast disease diagnosis.

[0032] An electronic device, comprising:

[0033] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, so that the one or more processors control the benign breast disease diagnosis system as described above to perform benign breast disease diagnosis

[0034] (3) Beneficial effects

[0035] The present invention provides a multimodal data-based system for diagnosing benign breast diseases, a storage medium, and an electronic device. Compared with existing technologies, the system has the following advantages:

[0036] In this paper, diagnostic information from images and text is effectively integrated, and the resulting fused feature vector is used to accurately classify and diagnose benign breast diseases, helping doctors make accurate clinical decisions. The proposed weighted fusion method not only improves diagnostic accuracy but also ensures the efficiency and reliability of the entire diagnostic process, making it suitable for clinical diagnosis of benign breast diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A structural block diagram of a multimodal data-based benign breast disease diagnosis system provided in an embodiment of the present invention;

[0039] Figure 2 A technical roadmap for diagnosing benign breast diseases based on multimodal data provided by an embodiment of the present invention;

[0040] Figure 3 A diagnostic flow chart based on a two-stage large model for diagnostic reporting provided by an embodiment of the present invention;

[0041] Figure 4 An example of a fine-tuning dataset format for the first stage of a two-stage large model provided in an embodiment of the present invention;

[0042] Figure 5 An example of a fine-tuning dataset format for the second stage of a two-stage large model provided in an embodiment of the present invention;

[0043] Figure 6 A diagnostic flowchart for ultrasound images based on multiple deep learning networks is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0045] The embodiments of the present application solve the technical problems of single technical means and insufficient integration of multimodal data by providing a benign breast disease diagnosis system, storage medium and electronic device based on multimodal data.

[0046] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0047] In related technologies, the comprehensive utilization of multimodal data (such as images and text) is still relatively insufficient, which limits the accuracy and comprehensiveness of diagnostic results. The failure to effectively integrate data from different modalities not only limits the performance of the diagnostic system, but also affects the overall judgment and prognostic analysis of the disease.

[0048] This embodiment of the present invention breaks away from traditional approaches that primarily rely on image data analysis, while leveraging the information contained in medical diagnostic reports (text data). The proposed multimodal data processing method can provide comprehensive diagnostic support for complex diagnostic tasks, enabling multi-angle and multi-level condition analysis.

[0049] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0050] Example 1:

[0051] like Figure 1 As shown, an embodiment of the present invention provides a system for diagnosing benign breast diseases based on multimodal data, comprising:

[0052] A raw data acquisition module is used to acquire and pre-process diagnostic reports and ultrasound images of patients with benign breast diseases;

[0053] A text data diagnosis prediction module is used to obtain a first diagnostic classification result based on a preset two-stage large model for the preprocessed diagnosis report; wherein the two-stage large model adopts a fine-tuned LLaMA-3-8B model;

[0054] An image data diagnosis prediction module is used to obtain a second diagnostic classification result based on a preset multiple deep learning network for the preprocessed ultrasound image;

[0055] The diagnosis conclusion acquisition module is used to quantize the first diagnosis classification result and the second diagnosis classification result respectively, and obtain the final diagnosis classification result after weighted fusion.

[0056] In this embodiment of the present invention, diagnostic information from images and text is effectively integrated, and the resulting fused feature vector is used to accurately classify and diagnose benign breast diseases, helping doctors make accurate clinical decisions. The proposed weighted fusion method not only improves diagnostic accuracy but also ensures the efficiency and reliability of the entire diagnostic process, making it suitable for clinical diagnosis of benign breast diseases.

[0057] like Figure 2 As shown, Figure 2 A technical roadmap for diagnosis of benign breast diseases based on multimodal data is given. Figure 2 Detailed introduction to each module of the above solution:

[0058] The raw data acquisition module is used to acquire and pre-process the diagnostic reports and ultrasound images of patients with benign breast diseases.

[0059] The preprocessing described in this module includes:

[0060] (1) Standardizing the processing of the diagnostic report, including text cleaning, removing irrelevant information, formatting the text structure, and adjusting the text length, to help improve the efficiency and accuracy of subsequent model processing of the text.

[0061] (2) Standardizing the ultrasound image, including image scaling and normalization, to help reduce the impact of noise, stabilize subsequent network training, and improve generalization capabilities.

[0062] As for the text data diagnosis prediction module, it is used to obtain the first diagnostic classification result for the preprocessed diagnosis report based on a preset two-stage large model; wherein the two-stage large model adopts the fine-tuned LLaMA-3-8B model.

[0063] See also Figure 3 and Figure 4 ,In the first stage, the LoRA method is used to fine-tune the q_proj layer and v_proj layer of the ,first LLaMA-3-8B model, and the preprocessed diagnosis report is used as ,the input of the fine-tuned first LLaMA-3-8B model to process the ,text data in a structured manner.

[0064] In the first stage, the LLaMA-3-8B model is fine-tuned to meet specific task requirements. To achieve this, the present invention employs LoRA (Low-Rank Adaptation) technology to fine-tune the model's q_proj and v_proj parameters. This technique significantly reduces the number of parameters while maintaining the model's original capabilities, thereby improving fine-tuning efficiency.

[0065] Note:

[0066] q_proj (Query Projection): In the self-attention mechanism, q_proj is responsible for projecting the input sequence into a query vector (Query). The query vector is used to perform a dot product operation with the key vector (Key) to calculate the attention score, which determines which parts of the input the model should focus on. Therefore, q_proj is crucial to how the model understands and parses the input information.

[0067] v_proj (Value Projection): v_proj projects the input sequence into value vectors. These value vectors are weighted and combined in the attention mechanism to generate the final attention output. These outputs determine the model's generation results, so v_proj directly affects the model's output quality.

[0068] Let the preprocessed diagnosis report be T. The process of converting it into structured data S by the fine-tuned model can be expressed as:

[0069] S=LLaMA-3-8B LoRA (T)

[0070] Among them, LLaMA-3-8B LoRA Represents the LLaMA-3-8B model fine-tuned by the LoRA method.

[0071] Through the above processing, the model outputs structured data S, which contains key information extracted from the original text, such as the diagnosis conclusion, lesion type, symptom description, etc. This structured data not only provides reliable input for subsequent classification tasks, but also greatly improves data processing efficiency.

[0072] Furthermore, the generated structured data not only serves as the foundation for subsequent classification predictions but also provides important intermediate steps, making the model's processing steps more transparent and explainable. By reviewing this structured data, doctors can understand how the model extracts and constructs information from unstructured text, which helps enhance confidence in clinical diagnoses.

[0073] See also Figure 4 and Figure 5 ,In the second stage, the LoRA method is used to fine-tune the q_proj layer and v_proj layer of the ,second LLaMA-3-8B model, and the structured text data is used as ,the input of the fine-tuned second LLaMA-3-8B model to obtain the ,first diagnostic classification result.

[0074] Based on the structured data generated in the first phase, the second phase aims to fine-tune the LLaMA-3-8B model to classify and predict the structured data and determine the specific type of benign breast disease. This step is crucial to the entire diagnostic process and directly affects diagnostic accuracy and clinical effectiveness.

[0075] Assume that the output of the first stage is structured data S, and the model output of the second stage is the disease classification result C. The process can be expressed as:

[0076] C=LLaMA-3-8B LoRA (S)

[0077] By classifying and predicting structured data, the model can output specific types of benign breast diseases, including breast adenosis, breast fibroadenoma, breast cyst, and intraductal papilloma. This classification result is directly used in clinical diagnosis, helping doctors make quick decisions and take appropriate treatment measures.

[0078] Furthermore, because classification predictions are based on the structured data generated in the previous stage, the entire diagnostic process has a clear interpretable chain. Doctors can trace every step from text to structured data to the final classification result, understanding how the model made its judgment, further enhancing the model's credibility.

[0079] In summary, the two-stage large model processing flow of this module combines the advantages of structured data generation and classification prediction, making the diagnosis of benign breast diseases more efficient and accurate.

[0080] Improved efficiency: In the first phase, the powerful capabilities of large models were leveraged to efficiently convert unstructured text into structured data, significantly reducing the time and complexity of manual processing.

[0081] Accurate information extraction: Through model fine-tuning, the system can accurately extract key information from the text and convert it into a structured form, ensuring the accuracy of subsequent classification.

[0082] Transparent classification process: The second-stage classification prediction is based on structured data, providing clear interpretability and making the entire diagnostic process more transparent and trustworthy.

[0083] Through two-stage processing, the present invention not only improves data processing efficiency but also significantly enhances the interpretability of diagnostic results. This method provides clinicians with a powerful support tool, making the diagnosis of benign breast diseases more accurate and efficient, and has broad application prospects.

[0084] As for the image data diagnosis and prediction module, it is used to obtain a second diagnostic classification result based on a preset multiple deep learning network for the preprocessed ultrasound image.

[0085] The overall algorithm structure and implementation process of this module are based on the synergy of multiple deep learning networks, aiming to improve the fine-grained classification and prediction capabilities of ultrasound images of benign breast diseases. Figure 6 As shown, the multiple deep learning networks include convolutional neural networks, multiple attention mechanisms, and Transformer modules; wherein the multiple attention mechanisms include SE modules, ECA modules, and CBAM modules in sequence.

[0086] For example, Figure 6 As shown, the convolutional neural network is ResNET34, and the second diagnostic classification result is obtained based on the preset multiple deep learning networks for the preprocessed ultrasound image; including:

[0087] (1) Preliminary feature extraction

[0088] The preprocessed ultrasound image is used as the input of the ResNET34 to extract preliminary features.

[0089] In the diagnosis of benign breast diseases, images often contain rich details, which are crucial in the early feature extraction stage. Preprocessed images are first passed through the ResNet34 network for preliminary feature extraction. ResNet34 is a deep convolutional neural network that effectively extracts local features of images. The preliminary features it outputs are represented by F:

[0090] F=ResNet34(X)

[0091] Where X is the pre-processed ultrasound image.

[0092] (2) Feature enhancement and attention mechanism

[0093] Ultrasound images of benign breast diseases typically have complex feature patterns, and traditional convolution operations may have difficulty fully capturing these patterns. Therefore, the embodiment of the present invention introduces the SE module, ECA module, and CBAM module to perform multi-level feature enhancement on the initially extracted feature map F, improving the expressive power of the feature map from multiple perspectives. The use of these modules can more accurately extract and amplify key features in the image, improving diagnostic accuracy. Among them:

[0094] SE module (Squeeze-and-Excitation Module): The SE module dynamically adjusts the weight of each channel by modeling the dependencies between channels to enhance important features. For ultrasound images of benign breast diseases, this adaptive adjustment between channels can highlight specific features related to the lesions and improve classification accuracy. The output feature map F of the SE module SEThe calculation formula is:

[0095] F SE =F·σ(W2·ReLU(W1·GlobalAvgPool(F)))

[0096] W1 and W2 are the weight matrices of the fully connected layer in the SE module, used for dimensionality reduction and channel dimension recovery, respectively. GlobalAvgPool is a global average pooling operation that averages the feature map into a single value in the spatial dimension. ReLU is an activation function used to introduce nonlinear features. σ is a Sigmoid activation function that constrains channel weights to the range [0, 1].

[0097] ECA module (Efficient Channel Attention Module): The ECA module achieves local interaction in the channel dimension through lightweight 1D convolution operations, improving feature expression capabilities. In particular, for subtle features in benign breast disease images, the ECA module can enhance the performance of these features without increasing computational complexity. The output feature map F of the ECA module ECA The calculation formula is:

[0098] F ECA =F·σ(Conv1D(GlobalAvgPool(F)))

[0099] Among them, Conv1D is a 1D convolution operation performed on the channel dimension for local interaction.

[0100] CBAM module (Convolutional Block Attention Module): The CBAM module further enhances the expressiveness of feature maps by combining channel attention and spatial attention. In ultrasound images of benign breast diseases, this dual attention mechanism can better capture complex image patterns and improve the robustness of diagnosis. The output feature map F of the CBAM module is CBAM The calculation formula is:

[0101] F CBAM =SpatialAttention(F·σ(ChannelAttention(F)))

[0102] Among them, ChannelAttention is a channel attention mechanism that uses global pooled features to generate channel weights; SpatialAttention is a spatial attention mechanism that captures the importance of features in the spatial dimension through convolution operations.

[0103] Finally, the output feature map F after each feature enhancement module isenhanced for:

[0104] F enhanced =F CBAM ·F ECA ·F SE

[0105] (3) Global feature modeling

[0106] Ultrasound images of benign breast diseases contain not only local features but also long-range dependencies that require understanding from a global context. While traditional convolutional neural networks excel at extracting local features, they have limitations when capturing global dependencies. Therefore, embodiments of the present invention introduce the Transformer module to capture these complex long-range dependencies.

[0107] The Transformer module uses the self-attention mechanism to enhance the feature map F enhanced Processing is performed to generate feature representations with global information. The processing flow includes the following steps:

[0108] 1) Feature flattening and transposition: Flatten the two-dimensional feature map F enhanced Flatten into a one-dimensional sequence and perform a transpose operation. The flattened feature representation is F flat :

[0109] F flat =Flatten(F enhanced )

[0110] Among them, Flatten is a flattening operation.

[0111] 2) Self-attention mechanism: The core of the Transformer module is the self-attention mechanism, whose formula is:

[0112]

[0113] Among them, Q (Query) is represented by the feature F flat The query matrix generated by linear transformation; K (Key) is represented by the feature F flat The key matrix generated; V(Value) is represented by the feature F flat The generated value matrix; d k is a scaling factor, generally the size of the dimensions of Q and K, to ensure numerical stability after the dot product operation; softmax is a soft maximization function used to normalize weights.

[0114] 3) Multi-head attention: In order to capture features at different spatial scales, Transformer uses a multi-head attention mechanism, the formula is:

[0115] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O

[0116] Among them, head i The feature representation output by each attention head; W O is the weight matrix of the linear transformation, which is used to map the multi-head attention output back to the original dimension; the global feature output after Transformer processing is represented as F transformer :

[0117] F transformer =Transformer(F flat )

[0118] (4) Classification prediction

[0119] Global feature representation F transformer After global average pooling, it is converted into a low-dimensional feature vector and finally classified and predicted through the fully connected layer. The classification output formula is:

[0120] y pred =softmax(W fc GlobalAvgPool(F transformer ))

[0121] Among them, W fc is the weight matrix of the fully connected layer, which maps the features to the classification space; GlobalAvgPool is the global average pooling operation for dimensionality reduction; y pred The probability distribution for classification predictions, expressed as the probability of each category.

[0122] By combining multiple deep learning techniques and attention mechanisms, the embodiments of the present invention can significantly improve the accuracy and robustness of image classification, reduce the risk of misdiagnosis, and demonstrate excellent potential in actual clinical applications.

[0123] The diagnosis conclusion acquisition module is used to respectively quantize the first diagnosis classification result and the second diagnosis classification result, and obtain the final diagnosis classification result after weighted fusion.

[0124] In the diagnosis of benign breast diseases, single-modality data analysis may not provide comprehensive information support. To achieve more accurate diagnosis, an embodiment of the present invention proposes a weighted fusion diagnosis method. This method integrates the output results of the image and text modalities through vectorization and weighted fusion, and uses a fully connected layer and a softmax classifier to make the final diagnosis prediction. This method leverages the advantages of different modalities and improves the accuracy and reliability of diagnosis through fusion.

[0125] Accordingly, the first diagnosis classification result and the second diagnosis classification result are quantized respectively, and a final diagnosis classification result is obtained after weighted fusion; including:

[0126] (1) Modal data vectorization

[0127] Text Data Vectorization: Breast ultrasound diagnosis text is used to generate prediction results using a two-stage large-scale model, the Benign Breast Disease Text Data Diagnosis Prediction Module. The text output is vectorized using the BERT model to generate a high-dimensional vector representation of the text.

[0128] Image Data Vectorization: Breast ultrasound images are used to generate predictions using a deep learning-based fine-grained benign breast disease image data diagnosis and prediction module. To facilitate fusion processing, the BERT model is also used to vectorize the predicted classification results, generating a high-dimensional vector representation of the image.

[0129] (2) Weighted Fusion

[0130] Weight selection: In the weighted fusion process, the weight coefficients of text and image are set to α = 0.6 and β = 0.4 respectively. This weight distribution is based on the following reasons:

[0131] Diagnostic text for benign breast diseases typically contains a wealth of diagnostic information, such as descriptions of symptoms and the doctor's initial assessment. This information is crucial for the final diagnosis. Therefore, the text is given a slightly higher weight, set to 0.6 for example. Ultrasound images, on the other hand, can directly display the morphology and characteristics of the lesion, which is crucial for diagnostic accuracy. However, because image features may contain some noise and ambiguity, their weight is slightly lower than that of text, set to 0.4 for example.

[0132] The corresponding weighted formula is:

[0133] v fused =0.6·v text +0.4·v image

[0134] Among them, v fused is the fusion vector, v text is the high-dimensional vector representation of the text, v image is a high-dimensional vector representation of the image.

[0135] (3) Final classification prediction

[0136] The fused vector v imageThe input vectors are fed into a fully connected layer, which then passes through a softmax layer to generate the final classification result. This method is simple and effective, suitable for integrated analysis of multimodal data. The fully connected layer maps the input fusion vector to the classification space, while the softmax layer generates a probability distribution for each class, predicting the final diagnosis result.

[0137] Thus far, the embodiment of the present invention has completed the entire process of diagnosing benign breast diseases based on multimodal data. It can be understood that the advantages of the proposed system include at least:

[0138] (1) Multimodal information integration: Through weighted fusion, the features of images and texts are effectively integrated to fully utilize the data advantages of different modalities.

[0139] (2) Improved accuracy: The fused feature representation can capture more information dimensions, thereby improving the accuracy and robustness of diagnosis.

[0140] (3) Simple and effective classification: The combination of fully connected layers and softmax classifiers ensures the efficiency and reliability of the diagnostic process and is suitable for rapid diagnosis in clinical practice.

[0141] Through a weighted fusion diagnosis method, this embodiment of the present invention effectively integrates image and text data for the diagnosis of benign breast diseases. This method fully utilizes information from both modalities. Through vectorization, weighted fusion, and simple classifier design, it achieves efficient and accurate diagnosis, providing a powerful support tool for clinical practice.

[0142] In order to further demonstrate the advantages of the embodiments of the present invention, the following specific examples are provided:

[0143] Example background:

[0144] During a routine physical examination, a patient's breast ultrasound examination revealed an abnormality. The doctor decided to use the multimodal weighted fusion diagnosis system of the present invention to conduct a comprehensive analysis of the patient's breast ultrasound image and related diagnostic text to obtain an accurate diagnosis result.

[0145] 1. Data Preparation

[0146] Image Data: The patient's breast ultrasound image is first input into the deep learning fine-grained benign breast disease image data diagnosis and prediction module. The image data undergoes preprocessing, including scaling, denoising, and normalization, to ensure consistency. The image then undergoes preliminary feature extraction using the ResNet34 network. Feature enhancement is further performed using the SE, ECA, and CBAM modules. Finally, the Transformer module performs global feature modeling to generate the final classification prediction. The image prediction results are vectorized using the BERT model to generate a high-dimensional vector representation.

[0147] Text Data: Unstructured text data, including patient medical histories, symptom descriptions, and preliminary diagnosis reports, is input into the two-stage large-scale model's text data diagnosis and prediction module for benign breast diseases. The text data is first preprocessed to remove irrelevant information and normalize it. In the first stage, the LLaMA-3-8B model structures this text data, extracts key information, and generates structured data. In the second stage, the model performs classification and prediction on this structured data, outputting diagnostic recommendations. The text prediction results are also vectorized using the BERT model to generate high-dimensional vector representations.

[0148] 2. Weighted Fusion and Diagnostic Prediction

[0149] Weighted Fusion: After obtaining the vectorized results for the image and text, the system assigns them different weights. Text data, because it contains richer diagnostic information, is given a higher weight (0.6), while image data, due to its intuitiveness and importance, is given a weight of 0.4. By weighted fusion of these two vectors, the system generates a composite feature vector that combines the visual information of the image and the diagnostic content of the text.

[0150] Final classification prediction: The fused comprehensive feature vector is used as input and processed by the fully connected layer. The fully connected layer maps the vector to the classification space, and then passes it through the Softmax layer to generate the final diagnosis. Based on the classification result, the system determines the patient's lesion type as breast adenosis and provides corresponding diagnostic recommendations.

[0151] 3. Example Summary

[0152] The multimodal weighted fusion diagnostic system effectively integrates diagnostic information from images and text, using the fused comprehensive feature vector to accurately predict disease classification. In this example, the system successfully diagnosed the patient's lesion as breast adenosis, helping doctors make accurate clinical decisions. This system's weighted fusion approach not only improves diagnostic accuracy but also ensures the efficiency and reliability of the entire diagnostic process, making it suitable for clinical diagnosis of benign breast diseases.

[0153] Example 2:

[0154] An embodiment of the present invention provides a storage medium storing a computer program for diagnosing benign breast diseases based on multimodal data, wherein the computer program enables a computer to control the benign breast disease diagnosis system as described in Example 1 to perform benign breast disease diagnosis.

[0155] Example 3:

[0156] An embodiment of the present invention provides an electronic device, including:

[0157] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, so that the one or more processors control the benign breast disease diagnosis system as described in Example 1 to perform benign breast disease diagnosis.

[0158] It is understandable that the storage medium and electronic device provided in the embodiments of the present invention correspond to the benign breast disease diagnosis system based on multimodal data provided in the embodiments of the present invention. The explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the benign breast disease diagnosis system and will not be repeated here.

[0159] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0160] 1. In this embodiment of the present invention, diagnostic information from images and text is effectively integrated, and the fused comprehensive feature vector is used to accurately classify and diagnose benign breast diseases, helping doctors make accurate clinical decisions. The proposed weighted fusion method not only improves diagnostic accuracy but also ensures the efficiency and reliability of the entire diagnostic process, making it suitable for clinical diagnosis of benign breast diseases.

[0161] 2. The two-stage large model processing flow proposed in the embodiment of the present invention combines the advantages of structured data generation and classification prediction, making the diagnosis of benign breast diseases more efficient and accurate.

[0162] 3. By combining multiple deep learning technologies and attention mechanisms, the embodiments of the present invention can significantly improve the accuracy and robustness of image classification, reduce the risk of misdiagnosis, and show excellent potential in actual clinical applications.

[0163] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0164] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A benign breast disease diagnosis system based on multimodal data, characterized in that: include: A raw data acquisition module is used to acquire and pre-process diagnostic reports and ultrasound images of patients with benign breast diseases; A text data diagnosis prediction module is used to obtain a first diagnostic classification result based on a preset two-stage large model for the preprocessed diagnosis report; wherein the two-stage large model adopts a fine-tuned LLaMA-3-8B model; An image data diagnosis prediction module is used to obtain a second diagnostic classification result based on a preset multiple deep learning network for the preprocessed ultrasound image; A diagnosis conclusion acquisition module, configured to quantize the first diagnosis classification result and the second diagnosis classification result respectively, and obtain a final diagnosis classification result after weighted fusion; The method of obtaining a first diagnostic classification result based on a preset two-stage large model for the pre-processed diagnostic report includes: In the first stage, the LoRA method is used to fine-tune the q_proj layer and v_proj layer of the first LLaMA-3-8B model, and the preprocessed diagnosis report is used as the input of the fine-tuned first LLaMA-3-8B model to structure the text data; In the second stage, the LoRA method is used to fine-tune the q_proj layer and v_proj layer of the second LLaMA-3-8B model, and the structured text data is used as the input of the fine-tuned second LLaMA-3-8B model to obtain the first diagnostic classification result.

2. The benign breast disease diagnosis system according to claim 1, wherein: The multiple deep learning networks include a convolutional neural network, a multiple attention mechanism, and a Transformer module; wherein the multiple attention mechanism includes an SE module, an ECA module, and a CBAM module in sequence.

3. The benign breast disease diagnosis system according to claim 2, wherein: The convolutional neural network is ResNET34; the second diagnostic classification result is obtained based on a preset multiple deep learning network for the preprocessed ultrasound image; including: The preprocessed ultrasound image is used as the input of the ResNET34 to extract preliminary features; The preliminary features are used as the input of the multi-attention mechanism, and global features are enhanced by the SE module, local features are enhanced by the ECA module, and spatial features are enhanced by the CBAM module to obtain an enhanced feature map; The enhanced feature map is used as the input of the Transformer module to model global features, and after global average pooling, the second diagnostic classification result is obtained through a fully connected layer.

4. The benign breast disease diagnosis system according to claim 1, wherein: The method of respectively quantizing the first diagnosis classification result and the second diagnosis classification result and obtaining the final diagnosis classification result after weighted fusion includes: Use the BERT model to vectorize the first diagnosis classification result and the second diagnosis classification result respectively to obtain a high-dimensional vector representation of the text and a high-dimensional vector representation of the image; weightedly fusing the high-dimensional vector representation of the text and the high-dimensional vector representation of the image to obtain a fused vector; The fusion vector is used as input and passes through a fully connected layer and a Softmax layer in sequence to obtain the final diagnosis classification result.

5. The benign breast disease diagnosis system according to claim 4, characterized in that: The fusion vector is expressed as: v fused =α·v text +β·v image Among them, v fused is the fusion vector, v text is the high-dimensional vector representation of the text, v image is the high-dimensional vector representation of the image, α and β are the corresponding weight coefficients, and α>β.

6. The benign breast disease diagnosis system according to any one of claims 1 to 5, characterized in that: The pretreatment includes: The diagnostic report is standardized, including text cleaning, removal of irrelevant information, formatting text structure and adjusting text length; and the ultrasound image is standardized, including image scaling and normalization.

7. A storage medium, characterized in that: The computer program for diagnosing benign breast diseases based on multimodal data is stored therein, wherein the computer program enables a computer to control the benign breast disease diagnosis system according to any one of claims 1 to 6 to perform benign breast disease diagnosis.

8. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, so that the one or more processors control the benign breast disease diagnosis system as described in any one of claims 1 to 6 to perform benign breast disease diagnosis.

Images