A disease recognition method and device based on multi-modal fusion

Abstract

The application discloses a disease recognition method and device based on multi-modal fusion, comprising the following steps: multi-modal feature extraction processing is performed on multi-modal original data, and quality and confidence of the multi-modal features are evaluated; based on an attention mechanism, dynamic weight generation processing is performed on the quality and confidence evaluation results of the multi-modal features, and adaptive feature fusion is performed on the features of each mode; first-stage preliminary screening processing is performed on the adaptive feature fusion results, and a heat map of an unusual case abnormal area or a common case diagnosis result is output; second-stage refined diagnosis is performed on the first-stage preliminary screening processing results in combination with a medical knowledge graph, and a refined diagnosis report is generated; in the first diagnosis stage, rapid screening positioning is realized, and in the second diagnosis stage, suspicious problems are analyzed in depth, so that it is explained which data are used for diagnosis, where the screening positioning result of the first diagnosis stage is observed, and what the second diagnosis stage identification reason is.

Description

Technical Field

[0001] This invention relates to the field of disease-aided diagnosis technology, and in particular to a disease identification method and apparatus based on multimodal fusion. Background Technology

[0002] With the rapid development of the medical field, the types of medical data used to assist doctors in diagnosing diseases are also increasing. These data include medical images (CT, MRI, X-ray), electronic medical records, pathological slides, genomics, etc. The sheer volume of medical data exceeds the limits of doctors' manual processing capabilities. Therefore, artificial intelligence technology is needed to assist doctors in processing some simpler medical data, allowing them to save time and energy to diagnose more diseases and help more patients.

[0003] Currently, artificial intelligence (AI) technology has matured, especially with the breakthroughs achieved in deep learning in areas such as image recognition and natural language processing. This allows AI to extract useful information from simple medical data, assisting doctors in processing it. However, current AI technology has some limitations when processing complex medical data, mainly in the following aspects: Data quality and annotation challenges: Medical data annotation is highly dependent on senior doctors, which is extremely costly and time-consuming. Subjective differences exist between different annotators, resulting in poor annotation consistency and serious annotation noise problems.

[0004] Data silos and privacy barriers: Difficulty in sharing data between hospitals creates data silos, resulting in insufficient data for model training. Strict patient privacy regulations make data acquisition and sharing extremely difficult.

[0005] Data bias: Training data may overrepresent a particular group (such as a specific region, ethnicity, or age), causing the model to perform significantly worse on unseen groups, resulting in reliability issues.

[0006] Lack of diagnostic evidence: There is only the diagnostic result, but it lacks information on the data used in the diagnosis, the screening and localization process, and the rationale for the diagnosis. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, this invention provides a disease identification method and apparatus based on multimodal fusion. It achieves rapid screening and localization of abnormal areas in uncommon cases or diagnostic results for common cases in the first diagnostic stage through multimodal quality and confidence assessment, dynamic multimodal fusion weighting, and two-stage hierarchical diagnosis. In the second diagnostic stage, it performs in-depth analysis and identification of the abnormal areas in uncommon cases' heat maps, outputting the final refined diagnostic results. It explains which data was used for diagnosis, where to look for the screening and localization results in the first diagnostic stage, and the reasons for the identification in the second diagnostic stage.

[0008] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: The first aspect of this application provides a disease identification method based on multimodal fusion, comprising the following steps: S101. Perform multimodal feature extraction processing on the original multimodal data, and evaluate the quality and confidence of the multimodal features; S102. Based on the attention mechanism, dynamic weight generation processing is performed on the evaluation results of multimodal feature quality and confidence, and adaptive feature fusion is performed on the features of each modality. S103. Perform the first-stage preliminary screening processing on the adaptive feature fusion results, and output a heat map of abnormal areas of uncommon cases or diagnostic results of common cases. S104. Based on the medical knowledge graph, conduct a second-stage refined diagnosis of the preliminary screening results from the first stage and generate a refined diagnostic report.

[0009] Furthermore, the multimodal raw data undergoes multimodal feature extraction processing, and the quality and confidence of the multimodal features are evaluated, including the following steps: Multimodal features are extracted from the original multimodal data using a modality-specific encoder. These multimodal features include visual features, text features, and temporal features. Add a multilayer perceptron to each modality-specific encoder; The quality and confidence scores of multimodal features are obtained by evaluating the quality and confidence of the multimodal features using a multilayer perceptron.

[0010] Furthermore, based on the attention mechanism, dynamic fusion weight generation processing is performed on the multimodal feature quality and confidence evaluation results, and adaptive feature fusion is performed on the features of each modality, including the following steps: The quality and confidence scores of multimodal features, along with the multimodal features themselves, are used as input data for a dynamic weight generation network. The input data is normalized using the Softmax function to obtain the dynamic weights of each mode, as shown in the following expression:

[0011] in, For visual feature weights, For text feature weights, For time-series feature weights, For function, For dynamic weight generation networks, The quality and confidence scores of visual features. The quality and confidence scores of the text features. For the quality and confidence scores of time-series features, As a visual feature, For text features, It is a time-series feature; The dynamic weights and features of each modality are adaptively fused using a weighted concatenation method to obtain the adaptive feature fusion results, the expressions of which are shown below:

[0012] in, For adaptive feature fusion results, For function, For visual feature weights, As a visual feature, For text feature weights, For text features, For time-series feature weights, It is a temporal characteristic.

[0013] Furthermore, the first-stage preliminary screening of the adaptive feature fusion results, and the output of heatmaps of abnormal areas or diagnostic results of common cases, includes the following steps: Anomaly detection is performed on the adaptive feature fusion results using a semantic segmentation network based on the U-Net architecture. Based on the labeled segmentation mask and the set loss function, the network parameters are continuously adjusted to locate the anomaly detection results and output a heat map of the abnormal area of ​​uncommon cases or the diagnosis results of common cases.

[0014] Furthermore, by combining the medical knowledge graph with the preliminary screening results from the first stage, a second stage of refined diagnosis is performed, generating a refined diagnostic report, including the following steps: By combining knowledge graphs with thresholding and connectivity analysis of heatmaps of abnormal regions in uncommon cases, the bounding box coordinates of all suspicious nodules were obtained. Based on the bounding box coordinates of all suspicious nodules, the feature blocks corresponding to each nodule region are precisely cropped from the multimodal fused feature map using 3D RoI Align; Each nodule's feature block is input into a classification network, which outputs the probability that each nodule is malignant and the classification confidence score.

[0015] Furthermore, the construction of a medical knowledge graph includes the following steps: Constructing a basic medical knowledge graph based on node type data and edge type data; The knowledge weights of the basic medical knowledge graph are adjusted based on the doctors' review feedback. By automatically parsing and extracting medical literature, and calculating the similarity and conflict between existing and newly extracted knowledge, it is determined whether the basic medical knowledge graph needs to be updated.

[0016] Optionally, the calculation method for the quality and confidence scores of the multimodal features in step S101 can be as follows: by using Bayesian deep learning or Monte Carlo Dropout, the variance of the multimodal features is calculated multiple times during inference, and the output variance of the multimodal features is used as a measure of confidence / uncertainty. If the variance of the multimodal features is larger, the quality and confidence of the multimodal features are lower, and vice versa.

[0017] Optionally, the dynamic fusion weight generation method in step S102 can be: based on the evaluation results of the multimodal feature quality and confidence using a gating mechanism, perform dynamic fusion weight generation processing for multimodal features, the expression of which is as follows:

[0018]

[0019]

[0020]

[0021] in, For dynamic fusion weights of multimodal features, For activation function, For visual feature weights, As a visual feature, For text feature weights, For text features, For time-series feature weights, As a time series feature, The quality and confidence scores of visual features. The quality and confidence scores of the text features. For the quality and confidence scores of time-series features, , , , , , These are learnable parameters.

[0022] Optionally, the preliminary screening method in the first stage of step S103 can be: inferring the abnormal regions in the adaptive feature fusion results by using class activation mapping or attention mining, and defining the presence of anomalies in the image by using image-level labeling, thereby reducing the cost of data annotation.

[0023] A second aspect of this application provides a disease recognition device based on multimodal fusion, comprising: The quality and confidence assessment module is used to extract multimodal features from the original multimodal data and to assess the quality and confidence of the multimodal features. The dynamic fusion weight module is used to dynamically generate weights for the evaluation results of multimodal feature quality and confidence based on the attention mechanism, and to perform adaptive feature fusion for the features of each modality. The first-stage diagnostic module is used to perform preliminary screening on the adaptive feature fusion results and output a heatmap of abnormal areas in uncommon cases or diagnostic results for common cases. The second-stage diagnostic module is used to combine medical knowledge graphs to perform a second-stage refined diagnosis on the preliminary screening results of the first stage, and generate a refined diagnostic report. Knowledge graph building unit, used to build and update medical knowledge graphs.

[0024] The beneficial effects of this application are: it enables rapid screening and localization of abnormal areas in uncommon cases or diagnostic results for common cases in the first diagnostic stage through multimodal quality and confidence assessment, multimodal dynamic fusion weighting, and two-stage hierarchical diagnosis; in the second diagnostic stage, it performs in-depth analysis and identification of the abnormal areas in uncommon cases through the heat map, outputs the final refined diagnostic results, explains which data were used for diagnosis, explains where to look for the screening and localization results in the first diagnostic stage, and explains the reasons for the identification in the second diagnostic stage.

[0025] The two-stage diagnostic architecture is an efficient computing resource allocator. The first diagnostic stage uses a lightweight model to quickly screen the entire map / data. Only a few suspicious areas require the use of a large model in the second diagnostic stage, which is computationally expensive, for in-depth analysis. Compared to the end-to-end approach that uses a complex model to process the entire data directly, this saves a lot of computing resources and speeds up the diagnostic process.

[0026] It provides multiple, intrinsic interpretable evidence, offering physicians intuitive evidence of why they made the diagnosis. This is reflected in the following aspects: data dependence (showing whether the diagnosis was mainly based on images or text), abnormal location maps (precisely indicating which area on the image is considered abnormal), and differential criterion (displaying key image features or text keywords that support the second-stage diagnostic conclusion), thereby breaking through the interpretability bottleneck and enhancing clinical credibility. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram illustrating the steps of a disease identification method based on multimodal fusion according to the present invention. Detailed Implementation

[0029] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0030] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0031] Example 1 A disease identification method based on multimodal fusion includes the following steps: S101. Perform multimodal feature extraction processing on the original multimodal data, and evaluate the quality and confidence of the multimodal features; Multimodal raw data can be obtained from the Hospital Information System (HIS / PACS), including medical imaging data, text data, and time-series data. Medical imaging data includes DICOM files from CT, MRI, X-ray, and Ultrasound scans. Text data includes structured electronic medical records (EMR), pathology reports, physician notes, discharge summaries, etc. Time-series data includes vital sign monitoring data and laboratory test result sequences. Multimodal feature extraction processing is performed on the multimodal raw data to obtain multimodal feature extraction results. These multimodal features include visual features, text features, and time-series features. Quality and confidence assessments of these multimodal features are then performed to obtain the multimodal feature quality and confidence assessment results.

[0032] The process of extracting multimodal features from raw multimodal data and evaluating the quality and confidence of these features includes the following steps: Multimodal features are extracted from the original multimodal data using a modality-specific encoder. These multimodal features include visual features, text features, and temporal features. Add a multilayer perceptron to each modality-specific encoder; The quality and confidence scores of multimodal features are obtained by evaluating the quality and confidence of the multimodal features using a multilayer perceptron.

[0033] For example, modality-specific encoders can be used to extract multimodal features from raw multimodal data. These encoders include image encoders, text encoders, and temporal data encoders. Image encoders (such as deep convolutional neural networks) can extract visual features from medical image data; text encoders (such as pre-trained language models) can extract text features from text data; and temporal data encoders (such as recurrent neural networks) can extract temporal features from temporal data. By attaching a lightweight multilayer perceptron to each modality-specific encoder and evaluating the quality and confidence of the visual, text, and temporal features respectively, quality and confidence scores for the multimodal features can be obtained.

[0034] Taking pulmonary nodule cases as an example, multimodal feature extraction processing is performed on the multimodal raw data of pulmonary nodule cases. Visual features of pulmonary nodules can be extracted from medical image data using an image encoder, yielding morphological, textural, and density features. Morphological features of pulmonary nodules represent volume, surface area, sphericity, etc. Textural features are represented by multi-scale texture entropy based on wavelet transform to capture the texture differences of blurred boundaries in ground-glass nodules. Density features are represented by the HU value distribution entropy of histograms to distinguish the density heterogeneity between solid and subsolid nodules. Text features can be extracted from pulmonary nodule text data using a text encoder. Normalized medical entity sequences (such as "20-year smoking history," "right upper lobe nodule," "3-month cough") are input into a "bidirectional LSTM + self-attention" model to capture the temporal correlation between "symptom-medical history-nodule features" (such as the potential correlation between "smoking history" and "malignancy risk of pulmonary nodules"), generating text features (F_T). The nodule features of the time series data sequence are encoded as sequence features (F_I), and combined with indicators such as "nodule segmentation contour smoothness and feature internal consistency" to generate a visual feature quality and confidence score (C_I). The higher the score, the more reliable the visual feature.

[0035] S102. Based on the attention mechanism, dynamic weight generation processing is performed on the evaluation results of multimodal feature quality and confidence, and adaptive feature fusion is performed on the features of each modality. The multimodal feature quality and confidence evaluation results are dynamically weighted by using an attention mechanism to obtain the dynamic weights of each modality. The generated dynamic weights of each modality and the features of each modality are then used to perform adaptive feature fusion by weighted concatenation to obtain the adaptive feature fusion result.

[0036] The dynamic fusion weight generation process based on the attention mechanism for evaluating the quality and confidence of multimodal features, and the adaptive feature fusion of features from each modality, includes the following steps: The quality and confidence scores of multimodal features, along with the multimodal features themselves, are used as input data for a dynamic weight generation network. The input data is normalized using the Softmax function to obtain the dynamic weights of each mode, as shown in the following expression:

[0037] in, For visual feature weights, For text feature weights, For time-series feature weights, For function, For dynamic weight generation networks, The quality and confidence scores of visual features. The quality and confidence scores of the text features. For the quality and confidence scores of time-series features, As a visual feature, For text features, It is a time-series feature; The dynamic weights and features of each modality are adaptively fused using a weighted concatenation method to obtain the adaptive feature fusion results, the expressions of which are shown below:

[0038] in, For adaptive feature fusion results, For function, For visual feature weights, As a visual feature, For text feature weights, For text features, For time-series feature weights, It is a temporal characteristic.

[0039] Taking pulmonary nodule cases as an example, the visual features, text features, and temporal features of pulmonary nodules, along with their corresponding quality and confidence scores, are used as input data for a dynamic weight generation network. The input data is normalized using the Softmax function to obtain the dynamic weights of the visual features, text features, and temporal features of pulmonary nodules. The visual features, text features, and temporal features of pulmonary nodules, along with their corresponding dynamic weights, are then adaptively fused using a weighted concatenation method to obtain the adaptive feature fusion results.

[0040] S103. Perform the first-stage preliminary screening processing on the adaptive feature fusion results, and output a heat map of abnormal areas of uncommon cases or diagnostic results of common cases. By using a semantic segmentation network to perform a preliminary screening of the adaptive feature fusion results, it is possible to efficiently detect whether there are any anomalies. Based on the labeled segmentation mask and the set loss function, the network parameters are continuously adjusted to locate the adaptive feature fusion results with anomalies and output a heatmap of the abnormal areas of uncommon cases, thereby enabling rapid screening of anomalies.

[0041] The first stage of preliminary screening of the adaptive feature fusion results, and the output of heatmaps of abnormal areas or diagnostic results of common cases, includes the following steps: Anomaly detection is performed on the adaptive feature fusion results using a semantic segmentation network based on the U-Net architecture. Based on the labeled segmentation mask and the set loss function, the network parameters are continuously adjusted to locate the anomaly detection results and output a heat map of the abnormal area of ​​uncommon cases or the diagnosis results of common cases.

[0042] Taking pulmonary nodule cases as an example, the U-Net semantic segmentation network is used to perform anomaly detection processing on the adaptive feature fusion results. It quickly extracts anomalous visual features, textual features, and temporal features with high quality and confidence scores from the pulmonary nodules. Based on these features, a preliminary judgment is made to determine whether the nodule is a common benign lesion (such as an inflammatory nodule or hamartoma). By setting a high recall threshold (e.g., 95%), the network model can directly output results for common cases with high quality and confidence scores, reducing the need for manual review. For rare cases with low quality and confidence scores, manual review is performed, thus balancing case processing efficiency and safety. Therefore, by directly outputting a preliminary diagnosis for cases identified as common diseases, the processing time for simple cases is significantly shortened (single-case inference time <100ms), allowing for a second-stage refined diagnosis for rare cases. By using labeled segmentation masks as supervision signals and Dice Loss + Focal Loss as loss functions, the model's ability to accurately locate nodules can be optimized. This enables the rapid location of heatmaps of abnormal regions in common or uncommon cases based on anomaly detection results, and outputs heatmaps of abnormal regions in uncommon cases or diagnostic results for common cases.

[0043] S104. Based on the medical knowledge graph, conduct a second-stage refined diagnosis of the preliminary screening results from the first stage and generate a refined diagnostic report. By utilizing heatmap thresholding and connected component analysis techniques, combined with knowledge graphs, suspicious bounding boxes can be screened in the heatmaps of abnormal regions in uncommon cases to obtain the bounding box coordinates of all suspicious nodules. Based on these bounding box coordinates, feature blocks corresponding to each nodule region are precisely cropped from the multimodal fusion feature map using 3D RoI Align technology. The feature blocks of each nodule are then input into a classification network to calculate the probability of malignancy of each nodule and its classification confidence, generating a refined diagnostic report, such as specific disease classification, malignancy probability, grade and type, along with a confidence score.

[0044] The construction of a medical knowledge graph includes the following steps: Constructing a basic medical knowledge graph based on node type data and edge type data; The knowledge weights of the basic medical knowledge graph are adjusted based on the doctors' review feedback. By automatically parsing and extracting medical literature, and calculating the similarity and conflict between existing and newly extracted knowledge, it is determined whether the basic medical knowledge graph needs to be updated.

[0045] For example, a basic knowledge graph can be constructed using node-type data and edge-type data. Node-type data includes lung nodule characteristics (morphology, density, location), clinical information (medical history, symptoms, laboratory indicators), disease type (lung adenocarcinoma, squamous cell carcinoma, inflammatory nodules, etc.), and treatment plans. Edge-type data includes associations (e.g., ground-glass nodules associated with lung adenocarcinoma) and evidence strength weights (converted from OR values ​​and confidence intervals in large-scale clinical studies into weighted values ​​of 0-1).

[0046] The knowledge weights are adjusted using reinforcement learning based on the doctor's review feedback (e.g., the association strength between ground-glass nodules + elevated CEA and lung adenocarcinoma should be adjusted from 0.7 to 0.85). The edge weights of the knowledge graph are used as the state of reinforcement learning, and the direction and magnitude of the adjustment based on the doctor's feedback are used as reward signals to train the policy network, which automatically learns how to optimize knowledge weights based on the doctor's feedback. For example, if the doctor repeatedly confirms the reliability of a certain association, the policy network will increase the update magnitude of the association's weight.

[0047] A multimodal literature parsing model is used to simultaneously parse text (e.g., "Lung nodules with diameter >10mm and spiculated appearance increase the risk of malignancy") and accompanying images (e.g., typical CT CT diagram of lung nodules). Knowledge triples of "lung nodule feature combination - association - disease type" are extracted (e.g., <diameter >10mm + spiculated appearance, association, lung adenocarcinoma). A graph attention network (GAT) is used to fuse existing knowledge graphs with newly extracted knowledge, calculating the similarity and conflict between old and new knowledge. The system automatically decides whether to add new nodes / edges or adjust existing edge weights (e.g., when new literature conflicts with existing knowledge, weights are assigned based on the new literature's impact factor and sample size to determine whether to update; if the new literature has a higher weight, existing knowledge is adjusted or new knowledge is added, i.e., the basic medical knowledge graph is updated; otherwise, no update is performed).

[0048] The second stage of refined diagnosis, which combines the results of the initial screening and processing in the first stage with a medical knowledge graph, and generates a refined diagnostic report, includes the following steps: By combining knowledge graphs with threshold processing and connected component analysis on the heatmaps of abnormal regions, the bounding box coordinates of all suspicious nodules are obtained. Based on the bounding box coordinates of all suspicious nodules, the feature blocks corresponding to each nodule region are precisely cropped from the multimodal fused feature map using 3D RoI Align; Each nodule's feature block is input into a classification network, which outputs the probability that each nodule is malignant and the classification confidence score.

[0049] Taking pulmonary nodule cases as an example, threshold processing and connected component analysis are applied to the heatmap of abnormal areas using a knowledge graph to capture finer-grained morphological, texture, and density features of pulmonary nodules, obtaining the bounding box coordinates of all suspicious pulmonary nodules. Based on the bounding box coordinates of all suspicious pulmonary nodules, feature blocks corresponding to each nodule region are precisely cropped from the multimodal fusion feature map using 3D RoI Align, combined with the knowledge graph (the association knowledge of "pulmonary nodule features - clinical information - disease type"). The feature blocks of each nodule are input into a classification network, which outputs the probability and classification confidence of each nodule as malignant, and generates a refined diagnostic report. The refined diagnostic report includes the probability of nodule malignancy and supporting evidence (such as the probability of malignancy of lung adenocarcinoma nodules, supporting evidence: lobulation sign, spiculation sign, smoking history).

[0050] Example 2 The difference from Example 1 is that, in a disease identification method based on multimodal fusion, the calculation method for the quality and confidence scores of multimodal features in step S101 can be as follows: by using Bayesian deep learning or Monte Carlo Dropout, the model calculates the variance of multimodal features multiple times during inference, and uses the output variance of multimodal features as a measure of confidence / uncertainty. If the variance of multimodal features is larger, the quality and confidence of multimodal features are lower, and vice versa.

[0051] Example 3 The difference from Embodiment 2 is that, in a disease identification method based on multimodal fusion, the dynamic fusion weight generation method in step S102 can be: based on a gating mechanism, the multimodal feature quality and confidence assessment results are processed to generate dynamic fusion weights for multimodal features, and the expression is as follows:

[0052]

[0053]

[0054]

[0055] in, For dynamic fusion weights of multimodal features, For activation function, For visual feature weights, As a visual feature, For text feature weights, For text features, For time-series feature weights, As a time series feature, The quality and confidence scores of visual features. The quality and confidence scores of the text features. For the quality and confidence scores of time-series features, , , , , , These are learnable parameters.

[0056] It should be noted that the gating mechanism can adopt the structural concept of gated cyclic unit (GRU) or gated linear unit (GLU) to design the gating mechanism, so as to control how much information is allowed to pass through by quality assessment score.

[0057] Example 4 The difference from Embodiment 3 is that the preliminary screening method in the first stage of step S103 can be: using only image-level labels (such as the presence of anomalies in the image), that is, using class activation mapping (CAM) or attention mining techniques to infer the approximate abnormal area of ​​the adaptive feature fusion result as input data for the second stage of refined diagnosis, which greatly reduces the cost of data annotation.

[0058] Example 5 The above is a disease identification method based on multimodal fusion provided in the embodiments of this application. The following is a disease identification device based on multimodal fusion provided in the embodiments of this application.

[0059] A disease identification device based on multimodal fusion, comprising: The quality and confidence assessment module is used to extract multimodal features from the original multimodal data and to assess the quality and confidence of the multimodal features. The dynamic fusion weight module is used to dynamically generate weights for the evaluation results of multimodal feature quality and confidence based on the attention mechanism, and to perform adaptive feature fusion for the features of each modality. The first-stage diagnostic module is used to perform preliminary screening on the adaptive feature fusion results and output a heatmap of abnormal areas in uncommon cases or diagnostic results for common cases. The second-stage diagnostic module is used to combine medical knowledge graphs to perform a second-stage refined diagnosis on the preliminary screening results of the first stage, and generate a refined diagnostic report. Knowledge graph building unit, used to build and update medical knowledge graphs.

[0060] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0061] The terms "first," "second," and "third," etc., used in this application's specification and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0062] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A disease identification method based on multimodal fusion, characterized in that, Includes the following steps: S101. Perform multimodal feature extraction processing on the original multimodal data, and evaluate the quality and confidence of the multimodal features; S102. Based on the attention mechanism, dynamic weight generation processing is performed on the evaluation results of multimodal feature quality and confidence, and adaptive feature fusion is performed on the features of each modality. S103. Perform the first-stage preliminary screening processing on the adaptive feature fusion results, and output a heat map of abnormal areas of uncommon cases or diagnostic results of common cases. S104. Based on the medical knowledge graph, conduct a second-stage refined diagnosis of the preliminary screening results from the first stage and generate a refined diagnostic report.

2. The disease identification method based on multimodal fusion according to claim 1, characterized in that, Step S101 includes the following steps: Multimodal features are extracted from the original multimodal data using a modality-specific encoder. These multimodal features include visual features, text features, and temporal features. Add a multilayer perceptron to each modality-specific encoder; The quality and confidence scores of multimodal features are obtained by evaluating the quality and confidence of the multimodal features using a multilayer perceptron.

3. The disease identification method based on multimodal fusion according to claim 1, characterized in that, Step S102 includes the following steps: The quality and confidence scores of multimodal features, along with the multimodal features themselves, are used as input data for a dynamic weight generation network. The input data is normalized using the Softmax function to obtain the dynamic weights of each mode, as shown in the following expression: in, For visual feature weights, For text feature weights, For time-series feature weights, For function, For dynamic weight generation networks, The quality and confidence scores of visual features. The quality and confidence scores of the text features. For the quality and confidence scores of time-series features, As a visual feature, For text features, It is a time-series feature; The dynamic weights and features of each modality are adaptively fused using a weighted concatenation method to obtain the adaptive feature fusion results, the expressions of which are shown below: in, For adaptive feature fusion results, For function, For visual feature weights, As a visual feature, For text feature weights, For text features, For time-series feature weights, It is a temporal characteristic.

4. The disease identification method based on multimodal fusion according to claim 1, characterized in that, Step S103 includes the following steps: Anomaly detection is performed on the adaptive feature fusion results using a semantic segmentation network based on the U-Net architecture. Based on the labeled segmentation mask and the set loss function, the network parameters are continuously adjusted to locate the anomaly detection results and output a heat map of the abnormal area of ​​uncommon cases or the diagnosis results of common cases.

5. The disease identification method based on multimodal fusion according to claim 1, characterized in that, Step S104 includes the following steps: By combining knowledge graphs with thresholding and connectivity analysis of heatmaps of abnormal regions in uncommon cases, the bounding box coordinates of all suspicious nodules were obtained. Based on the bounding box coordinates of all suspicious nodules, the feature blocks corresponding to each nodule region are precisely cropped from the multimodal fused feature map using 3D RoI Align; Each nodule's feature block is input into a classification network, which outputs the probability that each nodule is malignant and the classification confidence score.

6. The disease identification method based on multimodal fusion according to claim 1, characterized in that, The construction of the medical knowledge graph includes the following steps: Constructing a basic medical knowledge graph based on node type data and edge type data; The knowledge weights of the basic medical knowledge graph are adjusted based on the doctors' review feedback. By automatically parsing and extracting medical literature, and calculating the similarity and conflict between existing and newly extracted knowledge, it is determined whether the basic medical knowledge graph needs to be updated.

7. The disease identification method based on multimodal fusion according to any one of claims 1-6, characterized in that, The calculation method for the quality and confidence scores of the multimodal features in step S101 can be as follows: by using Bayesian deep learning or Monte Carlo Dropout, the variance of the multimodal features is calculated multiple times during inference, and the output variance of the multimodal features is used as a measure of confidence / uncertainty. If the variance of the multimodal features is larger, the quality and confidence of the multimodal features are lower, and vice versa.

8. The disease identification method based on multimodal fusion according to claim 7, characterized in that, The dynamic fusion weight generation method in step S102 can be as follows: based on the evaluation results of the quality and confidence of multimodal features using a gating mechanism, dynamic fusion weight generation processing of multimodal features is performed, and its expression is as follows: in, For dynamic fusion weights of multimodal features, For activation function, For visual feature weights, As a visual feature, For text feature weights, For text features, For time-series feature weights, As a time series feature, The quality and confidence scores of visual features. The quality and confidence scores of the text features. For the quality and confidence scores of time-series features, , , , , , These are learnable parameters.

9. The disease identification method based on multimodal fusion according to claim 8, characterized in that, The preliminary screening method in the first stage of step S103 can be: inferring the abnormal regions in the adaptive feature fusion result by class activation mapping or attention mining, and defining the image as abnormal by image-level labeling.

10. A disease identification device based on multimodal fusion, used to implement the disease identification method based on multimodal fusion as described in any one of claims 1-9, characterized in that, include: The quality and confidence assessment module is used to extract multimodal features from the original multimodal data and to assess the quality and confidence of the multimodal features. The dynamic fusion weight module is used to dynamically generate weights for the evaluation results of multimodal feature quality and confidence based on the attention mechanism, and to perform adaptive feature fusion for the features of each modality. The first-stage diagnostic module is used to perform preliminary screening on the adaptive feature fusion results and output a heatmap of abnormal areas in uncommon cases or diagnostic results for common cases. The second-stage diagnostic module is used to combine medical knowledge graphs to perform a second-stage refined diagnosis on the preliminary screening results of the first stage, and generate a refined diagnostic report. Knowledge graph building unit, used to build and update medical knowledge graphs.

Images