Breast benign disease diagnosis system based on multi-modal data
Through the integration and weighted fusion of multimodal data, the two-stage large model and multiple deep learning networks are used to solve the problem of insufficient single modal data in the diagnosis of benign breast diseases in the existing technology, and the accurate diagnosis of benign breast diseases is achieved. Improves the accuracy and reliability of the diagnosis.
Patent Information
- Application Number
- CN202411211429.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-30
AI Technical Summary
The existing computer-assisted diagnostic systems rely on single modal data in the diagnosis of benign breast diseases and cannot provide sufficient information support, affecting the accuracy and comprehensiveness of diagnostic results.
Using a multimodal data-based diagnosis system for benign breast diseases, the text is performed by acquiring and pre-processing of diagnostic reports and ultrasound images of patients with benign breast diseases, using two-stage large models and multiple deep learning networks, combining LoRA method and attention mechanism. Weighted fusion with image data to generate the final diagnostic classification results.
It improves the accuracy and reliability of the diagnosis of benign breast diseases, ensures the efficiency of the diagnostic process, is suitable for clinical diagnostic applications, and provides multi-angle and multi-level disease analysis.
Smart Images

Figure CN119028573A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer-aided diagnosis, and in particular to a benign breast disease diagnosis system, storage medium and electronic equipment based on multimodal data. Background Art
[0002] Computer-aided diagnosis (CAD) is an interdisciplinary technology that uses computer vision and artificial intelligence to process medical images (usually radiological and pathological images) to identify diseases.
[0003] Among the related technologies, computer-aided diagnosis systems have been gradually applied to the diagnosis of breast diseases. For example, automatic analysis of image data has been performed through technologies such as convolutional neural networks, and significant progress has been made in the classification of benign and malignant tumors and the subclassification of malignant tumors.
[0004] However, the research and application of benign breast diseases is still relatively weak. The existing deep learning technology in this field is mostly used to identify and classify malignant tumors, ignoring the refined diagnosis of benign lesions. In addition, the current diagnostic system mainly relies on image data for analysis. This single-modality analysis method may not provide sufficient information support when facing complex benign breast disease diagnosis, thus affecting the accuracy and comprehensiveness of the diagnostic results. Summary of the invention
[0005] 1. Technical issues to be solved
[0006] In view of the deficiencies in the prior art, the present invention provides a benign breast disease diagnosis system, storage medium and electronic device based on multimodal data, which solves the technical problems of single technical means and insufficient integration of multimodal data.
[0007] (II) Technical solution
[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:
[0009] A benign breast disease diagnosis system based on multimodal data, comprising:
[0010] A raw data acquisition module, used to acquire and pre-process diagnostic reports and ultrasound images of patients with benign breast diseases;
[0011] A text data diagnosis prediction module, for obtaining a first diagnosis classification result based on a preset two-stage large model for the preprocessed diagnosis report; wherein the two-stage large model adopts a fine-tuned LLaMA-3-8B model;
[0012] An image data diagnosis prediction module is used to obtain a second diagnosis classification result based on a preset multiple deep learning network for the preprocessed ultrasound image;
[0013] The diagnosis conclusion acquisition module is used to quantize the first diagnosis classification result and the second diagnosis classification result respectively, and obtain the final diagnosis classification result after weighted fusion.
[0014] Preferably, the method of obtaining a first diagnostic classification result based on a preset two-stage large model for the pre-processed diagnostic report includes:
[0015] In the first stage, the LoRA method is used to fine-tune the q_proj layer and v_proj layer of the first LLaMA-3-8B model, and the preprocessed diagnosis report is used as the input of the fine-tuned first LLaMA-3-8B model to process the text data in a structured manner;
[0016] In the second stage, the LoRA method is used to fine-tune the q_proj layer and v_proj layer of the second LLaMA-3-8B model, and the structured text data is used as the input of the fine-tuned second LLaMA-3-8B model to obtain the first diagnostic classification result.
[0017] Preferably, the multiple deep learning networks include convolutional neural networks, multiple attention mechanisms, and Transformer modules; wherein the multiple attention mechanisms include SE modules, ECA modules, and CBAM modules in sequence.
[0018] Preferably, the convolutional neural network is ResNET34; the second diagnostic classification result is obtained based on a preset multiple deep learning network for the preprocessed ultrasound image; including:
[0019] The preprocessed ultrasound image is used as the input of the ResNET34 to extract preliminary features;
[0020] Using the preliminary features as the input of the multi-attention mechanism, performing global feature enhancement through the SE module, performing local feature enhancement through the ECA module, and performing spatial feature enhancement through the CBAM module in sequence, to obtain an enhanced feature map;
[0021] The enhanced feature map is used as the input of the Transformer module to model the global features, and after global average pooling, the second diagnostic classification result is obtained through a fully connected layer.
[0022] Preferably, the step of quantizing the first diagnosis classification result and the second diagnosis classification result respectively and performing weighted fusion to obtain the final diagnosis classification result comprises:
[0023] Use the BERT model to vectorize the first diagnosis classification result and the second diagnosis classification result respectively, to obtain a high-dimensional vector representation of the text and a high-dimensional vector representation of the image;
[0024] Weightedly fusing the high-dimensional vector representation of the text and the high-dimensional vector representation of the image to obtain a fused vector;
[0025] The fusion vector is used as input and passes through a fully connected layer and a Softmax layer in sequence to obtain the final diagnosis classification result.
[0026] Preferably, the fusion vector is expressed as:
[0027] v fused =α·v text +β·v image
[0028] Among them, v fused is the fusion vector, v text is the high-dimensional vector representation of the text, v image is the high-dimensional vector representation of the image, α and β are the corresponding weight coefficients, and α>β.
[0029] Preferably, the pretreatment comprises:
[0030] The diagnostic report is standardized, including text cleaning, removal of irrelevant information, formatting text structure and adjusting text length; the ultrasound image is standardized, including image scaling and normalization.
[0031] A storage medium stores a computer program for diagnosing benign breast diseases based on multimodal data, wherein the computer program enables a computer to control the benign breast disease diagnosis system as described above to perform benign breast disease diagnosis.
[0032] An electronic device, comprising:
[0033] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, so that the one or more processors control the benign breast disease diagnosis system as described above to perform benign breast disease diagnosis
[0034] (III) Beneficial effects
[0035] The present invention provides a benign breast disease diagnosis system, storage medium and electronic device based on multimodal data. Compared with the prior art, it has the following beneficial effects:
[0036] In the present invention, the diagnostic information of images and texts is effectively integrated, and the fused comprehensive feature vector is used to accurately classify and diagnose benign breast diseases, helping doctors make accurate clinical decisions. The proposed weighted fusion method not only improves the accuracy of diagnosis, but also ensures the efficiency and reliability of the entire diagnostic process, and is suitable for clinical diagnosis of benign breast diseases. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0038] Figure 1 A structural block diagram of a benign breast disease diagnosis system based on multimodal data provided by an embodiment of the present invention;
[0039] Figure 2 A technical roadmap for diagnosis of benign breast diseases based on multimodal data provided by an embodiment of the present invention;
[0040] Figure 3 A diagnostic flow chart based on a two-stage large model for diagnostic reporting provided by an embodiment of the present invention;
[0041] Figure 4 An example of a fine-tuning dataset format for the first stage of a two-stage large model provided in an embodiment of the present invention;
[0042] Figure 5 An example of a fine-tuning dataset format for the second stage in a two-stage large model provided in an embodiment of the present invention;
[0043] Figure 6 A diagnostic flowchart based on multiple deep learning networks for ultrasound images is provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] The embodiments of the present application solve the technical problems of single technical means and insufficient integration of multimodal data by providing a benign breast disease diagnosis system, storage medium and electronic device based on multimodal data.
[0046] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:
[0047] In related technologies, the comprehensive use of multimodal data (such as images and text) is still relatively insufficient, which limits the accuracy and comprehensiveness of diagnostic results. Failure to effectively integrate data from different modalities not only limits the performance of the diagnostic system, but also affects the overall judgment and prognosis of the disease.
[0048] The embodiment of the present invention breaks through the traditional solution that mainly relies on the analysis of image data, and makes full use of the information contained in the medical diagnosis report (text data). The proposed multimodal data processing method can provide comprehensive diagnostic support when facing complex diagnostic tasks and realize multi-angle and multi-level disease analysis.
[0049] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0050] Embodiment 1:
[0051] like Figure 1 As shown, an embodiment of the present invention provides a benign breast disease diagnosis system based on multimodal data, comprising:
[0052] A raw data acquisition module, used to acquire and pre-process diagnostic reports and ultrasound images of patients with benign breast diseases;
[0053] A text data diagnosis prediction module, for obtaining a first diagnosis classification result based on a preset two-stage large model for the preprocessed diagnosis report; wherein the two-stage large model adopts a fine-tuned LLaMA-3-8B model;
[0054] An image data diagnosis prediction module is used to obtain a second diagnosis classification result based on a preset multiple deep learning network for the preprocessed ultrasound image;
[0055] The diagnosis conclusion acquisition module is used to quantize the first diagnosis classification result and the second diagnosis classification result respectively, and obtain the final diagnosis classification result after weighted fusion.
[0056] In the embodiment of the present invention, the diagnostic information of the image and text is effectively integrated, and the fused comprehensive feature vector is used to accurately classify and diagnose benign breast diseases, helping doctors make accurate clinical decisions. The proposed weighted fusion method not only improves the accuracy of diagnosis, but also ensures the efficiency and reliability of the entire diagnostic process, and is suitable for clinical diagnosis of benign breast diseases.
[0057] like Figure 2 As shown, Figure 2 A technical roadmap for diagnosis of benign breast diseases based on multimodal data is given. Figure 2 Detailed introduction to each module of the above solution:
[0058] The raw data acquisition module is used to acquire and pre-process the diagnostic reports and ultrasound images of patients with benign breast diseases.
[0059] The preprocessing described in this module includes:
[0060] (1) Standardizing the processing of the diagnostic report, including text cleaning, removing irrelevant information, formatting the text structure, and adjusting the text length, so as to help improve the efficiency and accuracy of subsequent model processing of the text.
[0061] (2) Standardizing the ultrasound image, including image scaling and normalization, to help reduce the impact of noise, stabilize subsequent network training, and improve generalization capabilities.
[0062] The text data diagnosis prediction module is used to obtain the first diagnosis classification result for the preprocessed diagnosis report based on a preset two-stage large model; wherein the two-stage large model adopts the fine-tuned LLaMA-3-8B model.
[0063] See also Figure 3 and Figure 4 ,In the first stage, the LoRA method is used to fine-tune the q_proj layer and v_proj layer of the first LLaMA-3-8B model, and the preprocessed diagnosis report is used as the input of the fine-tuned first LLaMA-3-8B model to process the text data in a structured manner.
[0064] In the first stage, the LLaMA-3-8B model is fine-tuned to adapt to specific task requirements. To achieve this goal, the embodiment of the present invention uses LoRA (Low-Rank Adaptation) technology to fine-tune the q_proj and v_proj of the model. The advantage of this technology is that it can significantly reduce the number of parameters while maintaining the original capabilities of the model, thereby improving the efficiency of fine-tuning.
[0065] Please note:
[0066] q_proj (Query Projection): In the self-attention mechanism, q_proj is responsible for projecting the input sequence into a query vector (Query). The query vector is used to perform a dot product operation with the key vector (Key) to calculate the attention score, thereby determining which parts of the input the model should focus on. Therefore, q_proj is crucial to how the model understands and parses the input information.
[0067] v_proj (Value Projection): v_proj is responsible for projecting the input sequence into value vectors. These value vectors are weighted and combined in the attention mechanism to generate the final attention output. These outputs determine the generation results of the model, so v_proj directly affects the output quality of the model.
[0068] Assume that the preprocessed diagnosis report is T, and the process of converting it into structured data S by the fine-tuned model can be expressed as:
[0069] S=LLaMA-3-8B LoRA (T)
[0070] Among them, LLaMA-3-8B LoRA Represents the LLaMA-3-8B model fine-tuned by the LoRA method.
[0071] Through the above processing, the structured data S output by the model contains the extraction of key information in the original text, such as diagnosis conclusion, lesion type, symptom description, etc. These structured data not only provide reliable input for subsequent classification tasks, but also greatly improve the efficiency of data processing.
[0072] Moreover, the generated structured data is not only the basis for subsequent classification predictions, but also provides important intermediate processes, making the model's processing steps more transparent and explainable. Doctors can understand how the model extracts and constructs information from unstructured text by viewing these structured data, which helps to enhance the trust of clinical diagnosis.
[0073] See also Figure 4 and Figure 5 ,In the second stage, the LoRA method is used to fine-tune the q_proj layer and v_proj layer of the second LLaMA-3-8B model, and the structured text data is used as the input of the fine-tuned second LLaMA-3-8B model to obtain the first diagnostic classification result.
[0074] Based on the structured data generated in the first phase, the goal of the second phase is to classify and predict these structured data by fine-tuning the LLaMA-3-8B model to determine the specific type of benign breast disease. This step is the key to the entire diagnostic process and directly affects the accuracy of diagnosis and clinical application effect.
[0075] Assume that the output of the first stage is structured data S, and the model output of the second stage is the disease classification result C. The process can be expressed as:
[0076] C=LLaMA-3-8B LoRA (S)
[0077] Through the classification and prediction of structured data, the model can output specific types of benign breast diseases, including breast adenosis, breast fibroadenoma, breast cyst, and intraductal papilloma. This classification result is directly used for clinical diagnosis, helping doctors to make quick judgments and take appropriate treatment measures.
[0078] Furthermore, since the classification prediction is based on the structured data generated in the previous stage, the entire diagnostic process has a clear interpretability chain. Doctors can trace every step from text to structured data to the final classification result to understand how the model makes judgments, which further enhances the credibility of the model.
[0079] In summary, the two-stage large model processing flow of this module combines the advantages of structured data generation and classification prediction, making the diagnosis of benign breast diseases more efficient and accurate.
[0080] Improved efficiency: In the first phase, the powerful capabilities of the big model were used to efficiently convert unstructured text into structured data, significantly reducing the time and complexity of manual processing.
[0081] Accurate information extraction: Through model fine-tuning, the system can accurately extract key information from the text and convert it into a structured form to ensure the accuracy of subsequent classification.
[0082] Transparent classification process: The second-stage classification prediction is based on structured data, which provides clear interpretability and makes the entire diagnosis process more transparent and credible.
[0083] Through two-stage processing, the embodiment of the present invention not only improves data processing efficiency, but also significantly enhances the interpretability of diagnostic results. This method provides a powerful support tool for clinicians, making the diagnosis of benign breast diseases more accurate and efficient, and has broad application prospects.
[0084] The image data diagnosis prediction module is used to obtain a second diagnostic classification result based on a preset multiple deep learning network for the preprocessed ultrasound image.
[0085] The overall algorithm structure and implementation process of this module are based on the synergy of multiple deep learning networks, aiming to improve the fine-grained classification prediction capability of ultrasound images of benign breast diseases. Figure 6 As shown, the multiple deep learning networks include convolutional neural networks, multiple attention mechanisms, and Transformer modules; wherein the multiple attention mechanisms include SE modules, ECA modules, and CBAM modules in sequence.
[0086] For example, Figure 6 As shown, the convolutional neural network is ResNET34, then for the preprocessed ultrasound image, based on the preset multiple deep learning networks, a second diagnostic classification result is obtained; including:
[0087] (1) Preliminary feature extraction
[0088] The preprocessed ultrasound image is used as the input of the ResNET34 to extract preliminary features.
[0089] In the diagnosis of benign breast diseases, images usually contain rich details, which are crucial in the early feature extraction stage. The preprocessed image is first subjected to preliminary feature extraction through the ResNet34 network. ResNet34 is a deep convolutional neural network that can effectively extract local features of images. The preliminary features of its output are represented by F:
[0090] F = ResNet34(X)
[0091] Where X is the preprocessed ultrasound image.
[0092] (2) Feature enhancement and attention mechanism
[0093] Ultrasound images of benign breast diseases usually have complex feature patterns, and traditional convolution operations may not be able to fully capture these patterns. Therefore, the embodiment of the present invention introduces the SE module, the ECA module, and the CBAM module to perform multi-level feature enhancement on the initially extracted feature map F, thereby improving the expressive power of the feature map from multiple angles. The use of these modules can more accurately extract and amplify key features in the image, thereby improving the accuracy of diagnosis. Among them:
[0094] SE module (Squeeze-and-Excitation Module): The SE module dynamically adjusts the weight of each channel and enhances important features by modeling the dependencies between channels. For ultrasound images of benign breast diseases, this adaptive adjustment between channels can highlight specific features related to the lesions and improve the accuracy of classification. The output feature map F of the SE module SEThe calculation formula is:
[0095] F SE =F·σ(W2·ReLU(W1·GlobalAvgPool(F)))
[0096] Among them, W1 and W2 are the weight matrices of the fully connected layer in the SE module, which are used for dimensionality reduction and channel dimension recovery respectively; GlobalAvgPool is a global average pooling operation, which averages the feature map into a single value in the spatial dimension; ReLU is an activation function used to introduce nonlinear features. σ is a Sigmoid activation function, which limits the channel weight to the range of [0,1].
[0097] ECA module (Efficient Channel Attention Module): The ECA module achieves local interaction in the channel dimension through lightweight 1D convolution operations to improve feature expression capabilities. In particular, for subtle features in benign breast disease images, the ECA module can enhance the performance of these features without increasing computational complexity. Output feature map F of the ECA module ECA The calculation formula is:
[0098] F ECA =F·σ(Conv1D(GlobalAvgPool(F)))
[0099] Among them, Conv1D is a 1D convolution operation performed on the channel dimension for local interaction.
[0100] CBAM module (Convolutional Block Attention Module): The CBAM module further enhances the expressiveness of the feature map by combining channel attention and spatial attention. In ultrasound images of benign breast diseases, this dual attention mechanism can better capture complex image patterns and improve the robustness of diagnosis. The output feature map F of the CBAM module CBAM The calculation formula is:
[0101] F CBAM =SpatialAttention(F·σ(ChannelAttention(F)))
[0102] Among them, ChannelAttention is a channel attention mechanism that uses global pooled features to generate channel weights; SpatialAttention is a spatial attention mechanism that captures the importance of features in the spatial dimension through convolution operations.
[0103] Finally, the output feature map F after each feature enhancement module isenhanced for:
[0104] F enhanced =F CBAM ·F ECA ·F SE
[0105] (3) Global feature modeling
[0106] Ultrasound images of benign breast diseases contain not only local features, but also long-range dependencies that need to be understood from the global context. Although traditional convolutional neural networks perform well in extracting local features, they have certain limitations in capturing global dependencies. Therefore, the Transformer module is introduced in the embodiment of the present invention to capture these complex long-range dependencies.
[0107] The Transformer module uses the self-attention mechanism to enhance the feature map F enhanced Processing is performed to generate feature representations with global information. The processing flow includes the following steps:
[0108] 1) Feature flattening and transposition: Flatten the two-dimensional feature map F enhanced Flatten into a one-dimensional sequence and perform a transpose operation. The flattened representation of the feature is F flat :
[0109] F flat =Flatten(F enhanced )
[0110] Among them, Flatten is a flattening operation.
[0111] 2) Self-attention mechanism: The core of the Transformer module is the self-attention mechanism, whose formula is:
[0112]
[0113] Among them, Q (Query) is represented by the feature F flat The query matrix generated by linear transformation; K(Key) is represented by the feature F flat The generated key matrix; V(Value) is represented by the feature F flat The generated value matrix; d k is a scaling factor, which is generally the dimension size of Q and K, ensuring the numerical stability after the dot product operation; softmax is a soft maximization function used to normalize the weights.
[0114] 3) Multi-head attention: In order to capture features at different spatial scales, Transformer uses a multi-head attention mechanism, the formula is:
[0115] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O
[0116] Among them, head i is the feature representation output by each attention head; W O is the weight matrix of the linear transformation, which is used to map the multi-head attention output back to the original dimension; the global feature output after Transformer processing is represented as F transformer :
[0117] F transformer = Transformer(F flat )
[0118] (4) Classification prediction
[0119] Global feature representation F transformer After global average pooling, it is converted into a low-dimensional feature vector and finally classified and predicted through the fully connected layer. The classification output formula is:
[0120] y pred =softmax(W fc GlobalAvgPool(F transformer ))
[0121] Among them, W fc is the weight matrix of the fully connected layer, which maps the features to the classification space; GlobalAvgPool is the global average pooling operation for dimensionality reduction; y pred The probability distribution for classification predictions, expressed as the probability of each class.
[0122] By combining multiple deep learning techniques and attention mechanisms, the embodiments of the present invention can significantly improve the accuracy and robustness of image classification, reduce the risk of misdiagnosis, and show excellent potential in actual clinical applications.
[0123] The diagnosis conclusion acquisition module is used to quantize the first diagnosis classification result and the second diagnosis classification result respectively, and obtain the final diagnosis classification result after weighted fusion.
[0124] In the diagnosis of benign breast diseases, data analysis of a single modality may not provide comprehensive information support. In order to make a more accurate diagnosis, the embodiment of the present invention proposes a weighted fusion diagnosis method, which integrates the output results of the two modalities of image and text through vectorization and weighted fusion, and uses a fully connected layer and a Softmax classifier for the final diagnosis prediction. This method takes advantage of the advantages of different modalities and improves the accuracy and reliability of diagnosis through fusion.
[0125] Accordingly, the first diagnosis classification result and the second diagnosis classification result are quantized respectively, and a final diagnosis classification result is obtained after weighted fusion; including:
[0126] (1) Modal data vectorization
[0127] Text data vectorization: The breast ultrasound diagnosis text generates prediction results through the two-stage large model breast benign disease text data diagnosis prediction module. The text output results are vectorized using the BERT model to generate a high-dimensional vector representation of the text.
[0128] Image data vectorization: Breast ultrasound images generate prediction results through the deep learning fine-grained breast benign disease image data diagnosis prediction module. In order to use the image output results for fusion processing, the BERT model is also used to vectorize the image prediction classification results to generate a high-dimensional vector representation of the image.
[0129] (2) Weighted Fusion
[0130] Weight selection: In the weighted fusion process, the weight coefficients of text and image are set to α = 0.6 and β = 0.4 respectively. This weight distribution is based on the following reasons:
[0131] Diagnostic texts for benign breast diseases usually contain rich diagnostic information, such as symptom descriptions and doctors' preliminary evaluations. This information is of great reference value for the final diagnosis. Therefore, the weight of the text is slightly higher, and is set to 0.6 for example. Ultrasound images can directly show the morphology and characteristics of the lesion area, which is crucial for the accuracy of diagnosis. However, since image features may have certain noise and ambiguity, the weight is slightly lower than that of text, and is set to 0.4 for example.
[0132] The corresponding weighted formula is:
[0133] v fused =0.6·v text +0.4·v image
[0134] Among them, v fused is the fusion vector, v text is the high-dimensional vector representation of the text, v image is a high-dimensional vector representation of the image.
[0135] (3) Final classification prediction
[0136] The fused vector v imageAs input, it enters the fully connected layer, and then generates the final classification result through the Softmax layer. This method is concise and effective, and is suitable for the integrated analysis of multimodal data. Among them, the fully connected layer is used to map the input fusion vector to the classification space; the Softmax layer is used to generate the probability distribution of each category and predict the final diagnosis result.
[0137] So far, the embodiment of the present invention has completed the entire process of diagnosing benign breast diseases based on multimodal data. It can be understood that the advantages of the proposed system include at least:
[0138] (1) Multimodal information integration: Through weighted fusion, the features of images and texts are effectively integrated to fully utilize the data advantages of different modalities.
[0139] (2) Improved accuracy: The fused feature representation can capture more information dimensions, thereby improving the accuracy and robustness of the diagnosis.
[0140] (3) Simple and effective classification: The combination of the fully connected layer and the Softmax classifier ensures the efficiency and reliability of the diagnostic process and is suitable for rapid diagnosis in clinical practice.
[0141] Through the weighted fusion diagnosis method, the embodiment of the present invention effectively integrates the image and text data in the diagnosis of benign breast diseases. This method makes full use of the information of the two modalities, and through vectorization, weighted fusion and simple classifier design, it achieves efficient and accurate diagnosis, providing a powerful support tool for clinical practice.
[0142] In order to further demonstrate the superiority of the embodiments of the present invention, the following specific examples are provided:
[0143] Example background:
[0144] During a routine physical examination, a breast ultrasound examination of a patient revealed an abnormality. The doctor decided to use the multimodal weighted fusion diagnosis system of the present invention to conduct a comprehensive analysis of the patient's breast ultrasound image and related diagnostic text to obtain an accurate diagnosis result.
[0145] 1. Data preparation
[0146] Image data: The patient's breast ultrasound image is first input into the deep learning fine-grained breast benign disease image data diagnosis prediction module. The image data is preprocessed, including scaling, denoising, and normalization, to ensure the consistency of the input image data. Then, the image is preliminarily feature extracted through the ResNet34 network, and further feature enhanced through the SE, ECA, and CBAM modules. Finally, the Transformer module performs global feature modeling to generate the final classification prediction results. The image prediction results are vectorized using the BERT model to generate a high-dimensional vector representation.
[0147] Text data: Unstructured text data such as patient medical history records, symptom descriptions, and preliminary diagnosis reports are input into the two-stage large model breast benign disease text data diagnosis and prediction module. The text data is first preprocessed to remove irrelevant information and perform standardization. In the first stage, the LLaMA-3-8B model structures the text data, extracts key information, and generates structured data. In the second stage, the model classifies and predicts the structured data and outputs diagnostic recommendations. The text prediction results are also vectorized by the BERT model to generate high-dimensional vector representations.
[0148] 2. Weighted fusion and diagnostic prediction
[0149] Weighted fusion: After obtaining the vectorized results of the image and text, the system assigns different weights to them. Text data is given a higher weight (0.6) because it contains richer diagnostic information, while image data is given a weight of 0.4 due to its intuitiveness and importance. The system generates a comprehensive feature vector by weighted fusion of the two vectors, which combines the visual information of the image and the diagnostic content of the text.
[0150] Final classification prediction: The fused comprehensive feature vector is used as input and enters the fully connected layer for processing. The fully connected layer maps the vector to the classification space, and then generates the final diagnosis result through the Softmax layer. Based on the classification result, the system determines that the patient's lesion type is breast adenosis and provides corresponding diagnostic suggestions.
[0151] 3. Example summary
[0152] Through the multimodal weighted fusion diagnosis system, the diagnostic information of images and texts can be effectively integrated, and the fused comprehensive feature vector can be used to make accurate disease classification predictions. In this example, the system successfully diagnosed the patient's lesion type as breast adenosis, helping doctors make accurate clinical decisions. The weighted fusion method of this system not only improves the accuracy of diagnosis, but also ensures the efficiency and reliability of the entire diagnostic process, and is suitable for clinical diagnosis of benign breast diseases.
[0153] Embodiment 2:
[0154] An embodiment of the present invention provides a storage medium storing a computer program for diagnosing benign breast diseases based on multimodal data, wherein the computer program enables a computer to control the benign breast disease diagnosis system as described in Example 1 to perform benign breast disease diagnosis.
[0155] Embodiment 3:
[0156] An embodiment of the present invention provides an electronic device, including:
[0157] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, so that the one or more processors control the benign breast disease diagnosis system as described in Example 1 to perform benign breast disease diagnosis.
[0158] It can be understood that the storage medium and electronic device provided in the embodiments of the present invention correspond to the benign breast disease diagnosis system based on multimodal data provided in the embodiments of the present invention, and the explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the benign breast disease diagnosis system, which will not be repeated here.
[0159] In summary, compared with the prior art, the present invention has the following beneficial effects:
[0160] 1. In the embodiment of the present invention, the diagnostic information of the image and text is effectively integrated, and the fused comprehensive feature vector is used to accurately classify and diagnose benign breast diseases, helping doctors make accurate clinical decisions. The proposed weighted fusion method not only improves the accuracy of diagnosis, but also ensures the efficiency and reliability of the entire diagnostic process, and is suitable for clinical diagnosis of benign breast diseases.
[0161] 2. The two-stage large model processing flow proposed in the embodiment of the present invention combines the advantages of structured data generation and classification prediction, making the diagnosis of benign breast diseases more efficient and accurate.
[0162] 3. By combining multiple deep learning technologies and attention mechanisms, the embodiments of the present invention can significantly improve the accuracy and robustness of image classification, reduce the risk of misdiagnosis, and show excellent potential in actual clinical applications.
[0163] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0164] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A benign breast disease diagnosis system based on multimodal data, It is characterized in that include: A raw data acquisition module, used to acquire and pre-process diagnostic reports and ultrasound images of patients with benign breast diseases; A text data diagnosis prediction module, for obtaining a first diagnosis classification result based on a preset two-stage large model for the preprocessed diagnosis report; wherein the two-stage large model adopts a fine-tuned LLaMA-3-8B model; An image data diagnosis prediction module is used to obtain a second diagnosis classification result based on a preset multiple deep learning network for the preprocessed ultrasound image; The diagnosis conclusion acquisition module is used to quantize the first diagnosis classification result and the second diagnosis classification result respectively, and obtain the final diagnosis classification result after weighted fusion.
2. The benign breast disease diagnosis system according to claim 1, It is characterized in that The method of obtaining a first diagnostic classification result based on a preset two-stage large model for the pre-processed diagnostic report includes: In the first stage, the LoRA method is used to fine-tune the q_proj layer and v_proj layer of the first LLaMA-3-8B model, and the preprocessed diagnosis report is used as the input of the fine-tuned first LLaMA-3-8B model to process the text data in a structured manner; In the second stage, the LoRA method is used to fine-tune the q_proj layer and v_proj layer of the second LLaMA-3-8B model, and the structured text data is used as the input of the fine-tuned second LLaMA-3-8B model to obtain the first diagnostic classification result.
3. The benign breast disease diagnosis system according to claim 1, It is characterized in that The multiple deep learning networks include convolutional neural networks, multiple attention mechanisms, and Transformer modules; wherein the multiple attention mechanisms include SE modules, ECA modules, and CBAM modules in sequence.
4. The benign breast disease diagnosis system according to claim 3, It is characterized in that The convolutional neural network is ResNET34; the second diagnostic classification result is obtained based on a preset multiple deep learning network for the preprocessed ultrasound image; including: The preprocessed ultrasound image is used as the input of the ResNET34 to extract preliminary features; Using the preliminary features as the input of the multi-attention mechanism, performing global feature enhancement through the SE module, performing local feature enhancement through the ECA module, and performing spatial feature enhancement through the CBAM module in sequence, to obtain an enhanced feature map; The enhanced feature map is used as the input of the Transformer module to model the global features, and after global average pooling, the second diagnostic classification result is obtained through a fully connected layer.
5. The benign breast disease diagnosis system according to claim 1, It is characterized in that The step of respectively quantizing the first diagnosis classification result and the second diagnosis classification result and obtaining the final diagnosis classification result after weighted fusion comprises: Use the BERT model to vectorize the first diagnosis classification result and the second diagnosis classification result respectively, to obtain a high-dimensional vector representation of the text and a high-dimensional vector representation of the image; Weightedly fusing the high-dimensional vector representation of the text and the high-dimensional vector representation of the image to obtain a fused vector; The fusion vector is used as input and passes through a fully connected layer and a Softmax layer in sequence to obtain the final diagnosis classification result.
6. The benign breast disease diagnosis system according to claim 5, It is characterized in that The fusion vector is expressed as: v fused =α·v text +β·v image Among them, v fused is the fusion vector, v text is the high-dimensional vector representation of the text, v image is the high-dimensional vector representation of the image, α and β are the corresponding weight coefficients, and α>β.
7. The benign breast disease diagnosis system according to any one of claims 1 to 6, It is characterized in that The pre-processing comprises: The diagnostic report is standardized, including text cleaning, removal of irrelevant information, formatting text structure and adjusting text length; the ultrasound image is standardized, including image scaling and normalization.
8. A storage medium, It is characterized in that It stores a computer program for diagnosing benign breast diseases based on multimodal data, wherein the computer program enables a computer to control the benign breast disease diagnosis system as described in any one of claims 1 to 7 to perform benign breast disease diagnosis.
9. An electronic device, It is characterized in that include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, so that the one or more processors control the benign breast disease diagnosis system as described in any one of claims 1 to 7 to perform benign breast disease diagnosis.
Citation Information
Patent Citations
Benign and malignant prediction method for multi-modal breast nodules and related device
CN114004798A
X-ray chest radiography diagnosis report generation method based on multi-task multi-mode deep learning
CN115223678A
Method, device and equipment for solving vocational education machine autonomous learning based on LLaMA model
CN116776855A
Question answering system based on large language model and knowledge graph, construction method of question answering system and intelligent data governance platform
CN117609470A
Multi-task image processing method and system based on breast cancer tumor multi-modal data
CN117635557A
Cited By
Colorectal cancer patient discovery system based on AI multi-mode technology
CN119230098A
Intelligent diagnosis auxiliary system and method for pediatric respiratory diseases
CN119400398A
Axis type spondyloarthritis auxiliary analysis system based on multi-modal data fusion
CN119763807A
Breast cancer detection method based on multi-modal fusion and three-stage prediction
CN120511031A
