Double-disease auxiliary diagnosis method and system based on multi-modal data fusion
By constructing a causal graph and graph neural network for multimodal data fusion, the accuracy problem in the diagnosis of comorbid infectious diseases and liver diseases was solved, and efficient diagnosis of dual diseases and personalized treatment recommendations were achieved.
Patent Information
- Application Number
- CN202610086772.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-22
- Publication Date
- 2026-05-01
AI Technical Summary
In the identification and synergistic treatment of comorbidities of infectious diseases and liver diseases, existing technologies struggle to accurately determine the primary cause, leading to low diagnostic accuracy for both diseases.
A multimodal data fusion-based approach was adopted to construct a fusion feature matrix. A causal graph of mechanism was established through temporal cross-information and Granger test. The diagnosis was performed by combining immune scores and graph neural networks, generating main symptom identification labels and recommending drug combinations.
It improves the accuracy of dual-disease diagnosis and the adaptability of treatment recommendations, enhances the interpretability of the system's diagnostic reasoning, and reduces the risk of drug interactions.
Smart Images
Figure CN121964108A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of disease diagnosis, and in particular relates to a dual-disease auxiliary diagnosis method and system based on multimodal data fusion. Background Technology
[0002] In clinical practice, the identification and synergistic treatment of comorbidities between infectious diseases and liver diseases have long faced practical challenges, including diagnostic confusion, unclear mechanisms, and difficulties in ensuring drug safety. Infectious diseases such as COVID-19 and monkeypox typically have acute onset and rapid inflammation, while liver diseases such as liver cancer and liver failure exhibit a trend of chronic progression and gradual decline in organ function. Although the two types of diseases differ in their physiological mechanisms and pathogenesis, they often present with highly overlapping clinical symptoms in comorbid situations. For example, changes in indicators such as fever, elevated liver enzymes, and immune abnormalities can be caused by either disease, making it difficult for traditional decision-making methods based on single modality or single etiology rules to accurately determine the primary cause, resulting in low accuracy in the diagnosis of dual diseases. Therefore, how to improve the accuracy of dual disease diagnosis has become an urgent technical problem to be solved. Summary of the Invention
[0003] The purpose of this invention is to design a dual-disease auxiliary diagnosis method and system based on multimodal data fusion, which can improve the accuracy of dual-disease diagnosis.
[0004] To achieve the above objectives, a dual-disease auxiliary diagnosis method based on multimodal data fusion is provided in a first aspect of the present invention, the method comprising: A fusion feature matrix is constructed based on the patient's imaging data, laboratory indicators, and medical history fields. A first indicator sub-matrix and a second indicator sub-matrix are extracted from the fusion feature matrix. The second indicator sub-matrix includes multiple time-series variables. The immune score is obtained by scoring based on the first index submatrix. Each of the time series variables is defined as a graph node, and the time cross-information value and the p-value of the Granger test between each graph node are calculated. If the time cross-information value is greater than the preset information value and the p value of the Granger test is less than the preset p value, then a directed edge is established between the corresponding graph nodes. The target edge weight of the directed edge is calculated based on the time cross-information value, the p-value of the Granger test, the immune score, and the preset path adjustment coefficient matrix; wherein, a mechanism causal graph is constructed based on the graph nodes and the target edge weight; The causal graph of the mechanism is input into a preset graph neural network to obtain a diagnostic graph vector. The diagnostic graph vector is then input into a preset fully connected neural network to obtain the main symptom identification label.
[0005] Furthermore, after inputting the diagnostic map vector into a preset fully connected neural network to obtain the main symptom identification label, the method further includes: The diagnostic map vector and the immune score are input into a preset drug recommendation model to obtain a drug recommendation score for each drug. Drugs with a drug recommendation score greater than a preset recommendation threshold are defined as recommended drugs.
[0006] Furthermore, after defining drugs with recommendation scores greater than a preset recommendation threshold as recommended drugs, the method further includes: Based on the immune score and a preset drug conflict matrix, a risk assessment is performed between each of the recommended drugs. If a risk is determined to exist, the corresponding recommended drug is defined as a conflicting drug pair.
[0007] Furthermore, the step of assessing the risk between each of the recommended drugs based on the immune score and a preset drug conflict matrix includes: The drug conflict matrix is used to determine whether there is a conflict between each of the recommended drugs. If a conflict is determined to exist and the immune score is greater than a preset score threshold, then a risk is determined to exist.
[0008] Furthermore, the construction of the fusion feature matrix based on the patient's imaging data, laboratory indicators, and medical history fields includes: Feature extraction is performed on the image data to obtain an image feature vector. The laboratory indicators are linearly interpolated to obtain a laboratory feature matrix. The medical history fields are encoded and compressed to obtain a medical history feature vector. The image feature vector and the medical history feature vector are copied and then concatenated with the laboratory feature matrix to obtain the fused feature matrix.
[0009] Further, the step of calculating the target edge weight of the directed edge based on the temporal cross-information value, the p-value of the Granger test, the immune score, and the preset path adjustment coefficient matrix includes: The initial edge weights of the directed edges are calculated based on the time cross-information value and the p-value of the Granger test. The target edge weight is obtained by multiplying the preset amplification factor, the immune score, and the path adjustment factor matrix, and adding the initial edge weight.
[0010] Further, the step of calculating the initial edge weight of the directed edge based on the time cross-information value and the p-value of the Granger test includes: The maximum value among all the aforementioned time-cross mutual information values is defined as the maximum cross mutual information value. The initial edge weights are obtained by dividing the time cross mutual information value by the maximum cross mutual information value and then multiplying it by the difference between the preset value and the p value of the Granger test.
[0011] In a second aspect, the present invention provides a dual-disease auxiliary diagnostic system based on multimodal data fusion, the system comprising: The acquisition unit is used to construct a fusion feature matrix based on the patient's imaging data, laboratory indicators, and medical history fields, and extract a first indicator sub-matrix and a second indicator sub-matrix from the fusion feature matrix respectively; wherein, the second indicator sub-matrix includes multiple time-series variables; The scoring unit is used to score according to the first index submatrix to obtain the immune score, define each of the time series variables as a graph node, and calculate the time cross-information value and the p-value of the Granger test between each of the graph nodes. A unit is established to establish directed edges between the corresponding graph nodes if the time cross-information value is greater than a preset information value and the p value of the Granger test is less than a preset p value. The calculation unit is used to calculate the target edge weight of the directed edge based on the time cross-information value, the p-value of the Granger test, the immune score, and the preset path adjustment coefficient matrix; wherein, a mechanism causal graph is constructed based on the graph nodes and the target edge weight; The identification unit is used to input the causal graph of the mechanism into a preset graph neural network to obtain a diagnostic graph vector, and input the diagnostic graph vector into a preset fully connected neural network to obtain a main symptom identification label.
[0012] In a third aspect of the invention, an electronic device is provided, the electronic device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the method described in the first aspect above.
[0013] In a fourth aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.
[0014] The beneficial technical effects of the present invention are at least as follows: To address the aforementioned issues, this invention provides a dual-disease auxiliary diagnostic method and system based on multimodal data fusion. Its core lies in employing a five-stage structured modeling strategy. First, by uniformly extracting chest and abdominal images, core laboratory indicators, and structured medical history, a time-aligned fusion feature matrix is constructed, serving as the standard input structure for the entire system. Second, an immune scoring model is built based on indicators such as transaminases, inflammatory factors, and viral load, generating a continuous quantitative score reflecting the current level of immune activity, which serves as a dynamic regulatory factor for downstream modules. In the mechanism modeling stage, key physiological variables are used as graph nodes. A causal graph of the pathological mechanism is constructed through mutual information and causal reasoning. The edge weights of specific transmission paths are adjusted in conjunction with the immune score, achieving dynamic mechanism regulation graph modeling with immunity as the mediating variable. In the primary symptom diagnosis stage, the model performs graph convolutional propagation based on this causal graph structure, outputting the classification result of the current dominant etiology and a mechanism-level semantic vector, providing diagnostic basis and structural features for the generation of treatment suggestions in the next stage. Finally, in the treatment generation module, the system integrates diagnostic graph vectors, primary symptom identification tags, and immune scores to generate recommended drug combinations for the current time point. Furthermore, by fusing knowledge of drug metabolism conflicts and immune status risk levels, it automatically identifies potentially dangerous combinations and issues warnings. This system, for the first time, establishes a multi-disease comorbidity modeling and decision-making path with immune status as the regulatory core, causal mechanism graphs as the structural expression, and individualized characteristics as the strategy foundation. This significantly enhances the system's interpretability of diagnostic reasoning and its adaptability to treatment recommendations. Attached Figure Description
[0015] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.
[0016] Figure 1 This is a flowchart of a dual-disease auxiliary diagnosis method based on multimodal data fusion provided in an embodiment of this application.
[0017] Figure 2 This is a schematic diagram of the structure of a dual-disease auxiliary diagnostic system based on multimodal data fusion provided in an embodiment of this application. Detailed Implementation
[0018] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0019] Please refer to Figure 1 , Figure 1This is a flowchart of a dual-disease auxiliary diagnosis method based on multimodal data fusion provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.
[0020] Step S101: Construct a fusion feature matrix based on the patient's imaging data, laboratory indicators, and medical history fields; extract the first indicator sub-matrix and the second indicator sub-matrix from the fusion feature matrix; wherein the second indicator sub-matrix includes multiple time-series variables. Step S102: Score according to the first index submatrix to obtain the immune score, define each time series variable as a graph node, and calculate the time cross-information value and the p-value of the Granger test between each graph node. Step S103: If the time cross-information value is greater than the preset information value and the p value of the Granger test is less than the preset p value, then a directed edge is established between the corresponding graph nodes. Step S104: Calculate the target edge weight of the directed edge based on the time cross-information value, the p-value of the Granger test, the immune score, and the preset path adjustment coefficient matrix; wherein, a mechanism causal graph is constructed based on the graph nodes and the target edge weight. Step S105: Input the mechanism causal graph into a preset graph neural network to obtain a diagnostic graph vector, and input the diagnostic graph vector into a preset fully connected neural network to obtain the main symptom identification label.
[0021] In step S101 of some embodiments, the aim is to construct a unified structured fusion feature matrix for the diagnosis and treatment of comorbid infectious diseases and liver diseases. This data is used for subsequent immune score modeling and causal graph structure analysis. Based on the clinical characteristics of these two disease types, this step selects only three strongly relevant data sources: patients' chest and abdominal imaging data, laboratory indicators (closely related to viral activity and liver function), and structured medical history fields. These data all originate from actual hospital information systems, including PACS (imaging), LIS (laboratory information system), and EMR (medical history), and have standardized acquisition mechanisms and unified interfaces.
[0022] First, the imaging data was limited to chest CT and abdominal CT / MRI, used to identify the degree of lung infection and changes in liver parenchyma. Images were retrieved from the PACS system in DICOM format, using the most recent combined chest and abdominal imaging result at the current time point. After windowing adjustments for the liver and lung windows, midline slices (approximately layers 15-18) were selected as input and fed into a ResNet-34 network for feature extraction. This network contains four residual modules, each with two 3×3 convolutional layers and 64-128-256-512 channels. The output feature maps were then subjected to global average pooling to obtain the image feature vector. It is used to characterize the manifestations of liver disease and lung lesions visible in images.
[0023] Secondly, among the laboratory indicators, seven parameters most representative of COVID-19 and liver disease were selected: ALT, AST, AFP, HBV-DNA, COVID-19 Ct value, CRP, and IL-6. These indicators were all extracted from the LIS system, sampled based on nearly 72 hours of historical data, with each time point being 6 hours. =12 time points. Linear interpolation is used to complete the data when the original data is incomplete. For example, if a patient has ALT records at hours 3, 6, and 9, but the record at hour 4 is missing, then the data is filled in using linear interpolation of the values from hours 3 and 6. After completion, a laboratory feature matrix is formed. Each row represents a point in time, and each column represents a specific indicator.
[0024] The medical history field is extracted directly from structured items in the electronic medical record, without using any free text. Selected fields include: Do you have a history of hepatitis B (yes / no)? Have you ever undergone radical liver cancer surgery? (Yes / No) Time since last surgery (months); Are you currently undergoing antiviral treatment (yes / no)?
[0025] These fields are output in a standardized format from the system's medical record entry page. They are directly converted into discrete labels and then fed into an embedding layer for encoding. Finally, they are compressed into a medical history feature vector using a two-layer perceptron (input dimension 4, hidden dimension 32, output dimension 64, ReLU activation). .
[0026] Image feature vector and medical history feature vector As static vectors, in order to construct the final time-aligned feature structure for subsequent diagnosis and modeling, the image feature vectors need to be... With medical history feature vector Mapping to a unified time series representation. Specifically, this involves copying these two static vectors. =12 times, so that it occupies the same dimension at each time point, and then compared with the time series laboratory feature matrix. Perform column concatenation. The final unified structured fusion feature matrix is as follows: ; in, To integrate the feature matrix, only indicators related to the two diseases are included, and irrelevant parameters such as blood glucose and blood pressure are removed. Through reasonable structural design, semantic unity and time alignment between different modalities are achieved, which has practical deployment feasibility and reproducibility. The laboratory feature matrix; Image feature vector; This is a feature vector representing medical history. For a total of length T Column vector, representing copying Second-rate; This indicates splicing along the column direction.
[0027] The laboratory feature matrix is Z-score standardized before splicing to make the values comparable; the static vectors are batch normalized during model training to achieve consistent distribution, so the overall tensor after splicing has structural stability and numerical consistency.
[0028] Furthermore, the feature matrix is fused. ,in, This represents 72 hours of historical data, divided into 6-hour time slices. Only data was extracted for establishing the immune score. The first indicator sub-matrix consists of five indicators that are highly correlated with immune activity. These five indicators include: CRP, IL-6, ALT, AST, and SARS-CoV-2 Ct value. All of these indicators are structured laboratory data, collected using the following methods: CRP and IL-6: Measured by blood tests, recorded every 24 hours; if missing, linear interpolation is used. ALT and AST: These are liver function transaminase indicators, recorded every 24 hours; Ct value: Extracted from the PCR nucleic acid test report; the lower the Ct value, the higher the viral load. In step S102 of some embodiments, scoring is performed based on the first indicator sub-matrix to obtain an immune score, specifically including: dividing the first indicator sub-matrix... The input is fed into a bidirectional GRU unit. The forward pass captures the rising trends of CRP and IL-6, while the backward pass captures short-term fluctuations such as the initial increase followed by a decrease in transaminase levels. The final layer's output is activated by a fully connected layer and a sigmoid function to obtain the immune score. As shown in the following formula: ; in, The final time node splicing output of the bidirectional GRU unit has a dimension of 128; For linear output layer weights, for transpose; For bias terms; This is the first index submatrix; This is the Sigmoid function, used to normalize score values.
[0029] To enhance the ability of the above scoring mechanism to distinguish etiological sensitivity, a regularization term derived from the cross-weights of infection-liver disease labels is introduced into the loss function during the model training phase. This regular expression term performs weighted suppression or activation based on whether "immune activation is dominated by viral changes" in the label. As shown in the following formula: ; in, The standard binary classification cross-entropy loss is used to train the immune score. Approximately expert scoring; is the regularization coefficient (set to 0.1); The samples were labeled with etiological tags by experts; This indicates the etiology attribution label for this sample (1 for infection-dominated, 0 for liver disease-dominated). This represents the partial derivative of the immune score with respect to the Ct value (virus-dominant influence). This represents the partial derivative of the immune score with respect to ALT (dominantly influenced by liver disease); this regularization term forces the model to learn the distinction between "infection elevation → immune activation" and "liver disease aggravation → suppression or fluctuation" through gradient comparison.
[0030] Immune score This can be interpreted as the current level of immune system activation. A higher score indicates a stronger correlation between inflammatory factors and liver damage markers, potentially placing the patient in the risk zone of an immune storm. For example, if a patient's IL-6 level rises from 28 pg / mL to 85 pg / mL in the past 24 hours, while their Ct value decreases from 28 to 21, and ALT remains above 180 U / L, the model outputs... =0.87, which is higher than the set threshold of 0.75, indicating a highly active state.
[0031] The above steps are based on the fusion feature matrix The key to constructing an individualized immune burden status scoring mechanism lies in quantifying the activation or suppression of the immune system at different time points in patients with comorbid infectious diseases and liver diseases, providing core regulatory variables for subsequent mechanism modeling and symptom diagnosis. Considering the complexity of diagnosis and treatment of infectious diseases (such as COVID-19 and monkeypox) and liver diseases (such as liver cancer and liver failure) in comorbid situations, their cross-effect mechanisms are often concentrated at the immune level. For example, viral infection can activate the immune response and induce liver damage, or cirrhosis can cause immunosuppression, leading to increased susceptibility to viruses. Therefore, this scoring mechanism has the following characteristics: it can identify the evolutionary differences between "infection-driven immune activation" and "liver disease-driven immunosuppression"; it can accurately perceive the dynamic changes of indicators over time; and it can output a dimensionless immune score that can be used for downstream graph structure regulation and risk inference.
[0032] Furthermore, from the fusion feature matrix Seven time-series variables closely related to the mechanism of transmission were extracted: ALT, AST, AFP (liver function related), HBV-DNA, SARS-CoV-2 Ct value (virus related), CRP, and IL-6 (immunity related), which constituted the second indicator sub-matrix. .
[0033] Treat each time series variable as a graph node. The feature vector of a graph node is the standardized value of that time-series variable at the last 12 time points, i.e. Together they form a node set All time-series variables are measured data or directly from [source name missing]. Column extraction, without secondary processing. Among them, Represents graph nodes The standardized values at the first time point, for example If ALT is a time-series variable, then This represents the ALT value at the first time point.
[0034] For any pair of graph nodes First, calculate its time cross-information value. (Window length is 6 hours, step size is 3 hours), and Granger causality test based on residuals is used to determine... Is this true? Time-crossing mutual information value. Used to measure two time series variables and The nonlinear coupling relationship between variables under time delay is investigated. Specifically, a sliding window approach is used for calculation, with a window length of six hours and a sliding step size of three hours. Within each window, the marginal probability distributions and joint probability distributions of the two variables are constructed using kernel density estimation, and the mutual information value is calculated accordingly. Finally, the average or maximum mutual information value across all time windows is taken as the variable pair. Time-interlacing mutual information It is used for statistical correlation judgment in the subsequent construction of causal graph edges.
[0035] Granger causality test is used to determine time series... Is it another time series? The causal variable is determined by comparing the residuals of two linear regression models to determine if there is a significant difference. First, a baseline model is constructed. ~ , ..., Then build an extended model ~ , ..., , , ..., If the extended model significantly reduces the sum of squared predicted residuals, then it is considered... Establishment, meaning yes The Granger causality test p-value can be obtained by constructing a linear regression model and using an F-test. Specifically, in implementation, the `statsmodels.tsa.stattools.grangercausalitytests` function in Python can be used, which internally performs the following process: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] , Construct a regression model with a lag order of I; compare using only... Self-hysteresis vs. usage and The model fit of the lagged values is evaluated; the F-test is used to calculate the significant difference in the residual variances between the two models, and the corresponding Granger test p-value is output. A smaller Granger test p-value indicates a better fit. right The stronger the predictive power, the stronger the causal basis. When the Granger test p-value < the preset p-value... When, explain The result is valid. The Granger test p-value is often referred to as the "Granger causal significance level" or "Granger significance probability".
[0036] In some embodiments, in steps S103 to S104, if And the Granger test p-value < Then the graph nodes Establish directed edges between them The maximum cross-information value among all time-based cross-information values is defined as the maximum cross-information value. The initial edge weights are obtained by dividing the time-based cross-information value by the maximum cross-information value and then multiplying it by the difference between the preset value and the p-value of the Granger test. As shown in the following formula: ; in, For graph nodes With graph nodes The time-crossing mutual information value between them; The maximum cross mutual information value is used for normalization; The p-value for the Granger test; This represents the initial edge weights, with a dimension of 7×7, and is not fully connected.
[0037] An immune score is introduced to modulate the initial edge weights. Since immune status directly affects the strength of the "virus to liver" transmission pathway, instead of uniform weighting across the entire graph, "etiological adjustment" is applied only to edges of specific pathway types to construct target edge weights. The target edge weight is obtained by multiplying the preset amplification coefficient, immune score, and path adjustment coefficient matrix together and adding the initial edge weight. As shown in the following formula: ; in, This is the magnification factor, which is empirically set to 0.5; It is an immune score; These are the initial edge weights; It is the path adjustment coefficient matrix, defined as follows: =+1, if These are virus-related indicators (HBV-DNA, Ct value). These are liver function indicators (ALT, AST, AFP), among which... For graph nodes Index; =-1, if These are indicators of liver function. These are immune markers (CRP, IL-6); =0, no adjustment; This design makes it possible to At higher levels, the side weights of the virus → liver damage pathway are enhanced, while the side weights of liver damage → immune indicators are suppressed, and vice versa, demonstrating the dynamic mechanism's weight regulation effect.
[0038] Furthermore, considering that individual differences and measurement errors may lead to instability in the single-point graph structure, after construction, all target edge weights constitute an edge weight matrix. A stability regularization term is added for smoothing, resulting in the final edge weight matrix. As shown in the following formula: ; in, = ; Weights for regularization terms; This is a Laplace smoothing term (Laplacian matrix) based on node similarity, used to reduce local edge weight oscillations caused by measurement noise; Ensure that the output edge weights of each node are normalized to meet the propagation requirements of the graph neural network.
[0039] The above steps construct a dynamic, interpretable causal graph structure. This method is used to capture the evolutionary patterns of causal relationships between key biomarkers associated with infectious diseases (such as COVID-19 and monkeypox) and liver diseases (such as liver cancer and liver failure). Compared to general graph structure modeling methods, it emphasizes three specific scenarios: First, there are interactions of immune pathways between the two diseases, requiring the modeling of a "bridging mechanism"; second, clinical data exhibit significant asynchronous dynamics (such as CRP rising first and ALT lagging), making it impossible to simply use static graphs; and third, individual immune status (already modeled as a score in step two). This will significantly affect the strength of the interaction between indicators, therefore the graph structure cannot be fixed, but should be dynamically adjusted.
[0040] In step S105 of some embodiments, the aim is to base the mechanism on the causal graph structure. Perform a primary symptom diagnosis model and output a primary symptom identification label. This characterizes the dominant disease attribution of the current patient's condition in a comorbidity scenario of infectious disease and liver disease (e.g., "dominated by COVID-19" or "dominated by liver failure"). The causal graph of the mechanism is input into a pre-defined graph neural network to obtain a diagnostic graph vector. This diagnostic graph vector is then input into a pre-defined fully connected neural network to obtain the primary symptom identification label.
[0041] Specifically, to complete the modeling for the main symptom diagnosis, a lightweight graph neural network structure (two layers of graph convolutional units) based on a graph structure propagation mechanism is adopted. The core idea is to utilize graph structures. This model characterizes the "mechanism propagation path" between different indicators and identifies the "dominant pathological path" in the structure through node state propagation. Unlike the traditional method of "feature splicing + fully connected classification", this model performs propagation calculations entirely on a structured graph space, which has strong mechanism explanatory power and is consistent with the "mechanism graph modeling" direction emphasized in this invention.
[0042] The first layer graph convolution propagation takes the following form: ; in, For the first The intermediate state of a node after propagation at the first layer; For the mechanism causal diagram, the first The target edge weight; This indicates the node state of the graph in the initial input layer; This indicates the corresponding bias parameter; Learnable weights for the node feature propagation layer (using 1-dimensional weights since each node only has its current value); This indicates graph edge normalization to avoid propagation bias caused by degree imbalance; ReLU is the standard nonlinear activation.
[0043] After the first layer of propagation, we obtain the intermediate state updated by each node. Then, a second layer of graph convolution is used to perform full graph aggregation, constructing a graph-level diagnostic graph vector. As input features for primary symptom diagnosis: ; in, The attention weights are learnable values, representing the node's attention weights. Contribution to the current diagnostic results; attention mechanism by definition, Weight vector and The parameters to be trained, The attention score for a node in the current graph structure and mechanism propagation state is used to measure the importance of the node in participating in the construction of the graph-level representation. This is a bias parameter used in attention scoring calculations to adjust the baseline level of the attention mapping function.
[0044] Diagnostic vector graphics The input is fed into a two-layer fully connected neural network (MLP), and the output is a binary classification label for the primary symptom. , =0 indicates that liver disease is the dominant factor. =1 indicates that infectious diseases are the dominant factor.
[0045] In practical deployment, the model is trained in a supervised manner, with training data consisting of "primary symptom labels" determined by doctors in real historical cases and corresponding laboratory indicators (extracted from a structured electronic medical record system to ensure consistency). To enhance the model's ability to identify "mechanism propagation paths," an additional graph structure preservation regularization term is introduced into the training loss. This ensures that the differences in node features on high-weighted paths are better preserved during training, thereby enhancing the effectiveness of structure-driven diagnosis.
[0046] For example, if a patient currently has significantly elevated ALT (380 U / L), AST (420 U / L), and AFP (120 ng / mL), while HBV-DNA and Ct values remain relatively unchanged, and IL-6 is slightly elevated, then the propagation weight between the three liver function nodes in the graph structure is relatively large, ultimately... The vector activates the "liver disease-dominant" path in the diagnostic module, outputting... =0.
[0047] Steps S101 to S105 of this embodiment involve constructing a fusion feature matrix based on the patient's imaging data, laboratory indicators, and medical history fields, and extracting a first indicator sub-matrix and a second indicator sub-matrix from the fusion feature matrix. The second indicator sub-matrix includes multiple temporal variables. An immune score is obtained by scoring based on the first indicator sub-matrix. Each temporal variable is defined as a graph node, and the temporal cross-information value and the p-value of the Granger test are calculated between each graph node. If the temporal cross-information value is greater than a preset information value and the p-value of the Granger test is less than a preset p-value, a directed edge is established between the corresponding graph nodes. The target edge weight of the directed edge is calculated based on the temporal cross-information value, the p-value of the Granger test, the immune score, and a preset path adjustment coefficient matrix. A causal graph is constructed based on the graph nodes and the target edge weights. The causal graph is input into a preset graph neural network to obtain a diagnostic graph vector, which is then input into a preset fully connected neural network to obtain the main symptom identification label. This improves the accuracy of dual-disease diagnosis.
[0048] In some embodiments, after step S105, based on the main symptom identification tag and diagnostic vector Integrating current immune scores This generates a well-structured and reproducible personalized treatment recommendation model to calculate the most appropriate recommended drug at the current point in time. and output a set of potential conflicting drug pairs. .
[0049] To complete the modeling of personalized treatment combinations, a two-stage drug recommendation model is constructed. The first stage is a protocol generator, which takes the current diagnostic vector as input. and immune score Output drug recommendation score vector ,in for The The nth component represents the nth component. The model calculates a drug recommendation score for a given drug. The model structure is a two-layer feedforward neural network; the first layer contains 64 hidden units, and the second layer outputs a dimension of... The model uses ReLU and Sigmoid activation functions, and is supervised and aligned with real drug labels (historical drug combinations labeled by doctors) during training. The model structure is as follows:
[0050] in, The diagnostic vector and the immune score are concatenated. This represents the weight matrix of the first fully connected layer of the drug recommendation model. This represents the weight matrix of the second fully connected layer network. Indicates the bias parameters of the first layer fully connected network. This represents the bias parameter of the second-layer fully connected network; For the Sigmoid function, each output This can be interpreted as the probability of drug selection. Drug recommendations with a score greater than a preset recommendation threshold are considered. The drug is defined as a recommended drug. The second stage is a conflict filtering mechanism. Because there may be metabolic cascade effects between drug combinations (e.g., both drugs are metabolized by CYP3A4, or they have synergistic immune activation effects), medication decisions cannot be made solely based on drug recommendation scores. Therefore, a structured rule module is introduced in the output stage, based on the current immune score. Drug Conflict Matrix Determine if any high-risk combinations exist, adjust the recommendations accordingly, and formulate recommended medications. Drug interactions .
[0051] The drug conflict matrix is used to determine whether there is a conflict between each recommended drug. If a conflict is determined to exist and the immune score is greater than the preset score threshold, then a risk is identified and the corresponding recommended drugs are defined as conflicting drug pairs.
[0052] The overall judgment logic is: for any pair of drugs ,like , , and If the drug pair is deemed unsafe under the current state of immune activation, then the conflicting drug pair should be added. .in, The preset recommended threshold is generally set to 0.5; The scoring threshold is typically set to 0.7; if Lower (e.g.) If the value is <0.3, the treatment for conflicting drug pairs is relaxed; otherwise, strict restrictions are imposed. Ultimately, the recommended drugs are... Conflicting drugs .in, Indicates drug There is a conflict. Indicates drug There is no conflict.
[0053] For example, if the patient's current diagnosis is COVID-19 dominant ( =1), diagnostic map vector The proportion of viral indicators is the main factor, and the immune score is [missing information]. =0.81 (relatively high), the model output recommends "lopinavir" and "sorafenib" as drugs, because these two have metabolic competition ( =1), and the current immune score is high, then the system outputs ={"Lopinavir","Sorafenib"}, and output the following: ={("Lopinavir","Sorafenib")}, for doctors' reference.
[0054] Please see Figure 2 This application also provides a dual-disease auxiliary diagnosis system based on multimodal data fusion, which can realize the above-mentioned dual-disease auxiliary diagnosis method based on multimodal data fusion. The system includes: The acquisition unit 201 is used to construct a fusion feature matrix based on the patient's imaging data, laboratory indicators, and medical history fields, and extract a first indicator sub-matrix and a second indicator sub-matrix from the fusion feature matrix; wherein, the second indicator sub-matrix includes multiple time-series variables; Scoring unit 202 is used to score according to the first index submatrix to obtain the immune score, define each time series variable as a graph node, and calculate the time cross-information value and the p-value of the Granger test between each graph node. Establish unit 203, which is used to establish directed edges between the corresponding graph nodes if the time cross-information value is greater than the preset information value and the p value of the Granger test is less than the preset p value; The calculation unit 204 is used to calculate the target edge weight of the directed edge based on the time cross-information value, the p-value of the Granger test, the immune score, and the preset path adjustment coefficient matrix; wherein, a mechanism causal graph is constructed based on the graph nodes and the target edge weight; The recognition unit 205 is used to input the mechanism causal graph into a preset graph neural network to obtain a diagnostic graph vector, and input the diagnostic graph vector into a preset fully connected neural network to obtain the main symptom recognition label.
[0055] The specific implementation of this dual-disease auxiliary diagnosis system based on multimodal data fusion is basically the same as the specific implementation of the dual-disease auxiliary diagnosis method based on multimodal data fusion described above, and will not be repeated here.
[0056] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A dual-disease auxiliary diagnostic method based on multimodal data fusion, characterized in that, The method includes: A fusion feature matrix is constructed based on the patient's imaging data, laboratory indicators, and medical history fields. A first indicator sub-matrix and a second indicator sub-matrix are extracted from the fusion feature matrix. The second indicator sub-matrix includes multiple time-series variables. The immune score is obtained by scoring based on the first index submatrix. Each of the time series variables is defined as a graph node, and the time cross-information value and the p-value of the Granger test between each graph node are calculated. If the time cross-information value is greater than the preset information value and the p value of the Granger test is less than the preset p value, then a directed edge is established between the corresponding graph nodes. The target edge weight of the directed edge is calculated based on the time cross-information value, the p-value of the Granger test, the immune score, and the preset path adjustment coefficient matrix; wherein, a mechanism causal graph is constructed based on the graph nodes and the target edge weight; The causal graph of the mechanism is input into a preset graph neural network to obtain a diagnostic graph vector. The diagnostic graph vector is then input into a preset fully connected neural network to obtain the main symptom identification label.
2. The dual-disease auxiliary diagnosis method based on multimodal data fusion according to claim 1, characterized in that, After inputting the diagnostic map vector into a preset fully connected neural network to obtain the main symptom identification label, the method further includes: The diagnostic map vector and the immune score are input into a preset drug recommendation model to obtain a drug recommendation score for each drug. Drugs with a drug recommendation score greater than a preset recommendation threshold are defined as recommended drugs.
3. The dual-disease auxiliary diagnosis method based on multimodal data fusion according to claim 2, characterized in that, After defining drugs with recommendation scores greater than a preset recommendation threshold as recommended drugs, the method further includes: Based on the immune score and a preset drug conflict matrix, a risk assessment is performed between each of the recommended drugs. If a risk is determined to exist, the corresponding recommended drug is defined as a conflicting drug pair.
4. The dual-disease auxiliary diagnosis method based on multimodal data fusion according to claim 3, characterized in that, The step of assessing the risk between each recommended drug based on the immune score and a preset drug conflict matrix includes: The drug conflict matrix is used to determine whether there is a conflict between each of the recommended drugs. If a conflict is determined to exist and the immune score is greater than a preset score threshold, then a risk is determined to exist.
5. The dual-disease auxiliary diagnosis method based on multimodal data fusion according to claim 1, characterized in that, The construction of a fusion feature matrix based on the patient's imaging data, laboratory indicators, and medical history fields includes: Feature extraction is performed on the image data to obtain an image feature vector. The laboratory indicators are linearly interpolated to obtain a laboratory feature matrix. The medical history fields are encoded and compressed to obtain a medical history feature vector. The image feature vector and the medical history feature vector are copied and then concatenated with the laboratory feature matrix to obtain the fused feature matrix.
6. The dual-disease auxiliary diagnosis method based on multimodal data fusion according to claim 1, characterized in that, The step of calculating the target edge weight of the directed edge based on the time cross-information value, the p-value of the Granger test, the immune score, and the preset path adjustment coefficient matrix includes: The initial edge weights of the directed edges are calculated based on the time cross-information value and the p-value of the Granger test. The target edge weight is obtained by multiplying the preset amplification factor, the immune score, and the path adjustment factor matrix, and adding the initial edge weight.
7. The dual-disease auxiliary diagnosis method based on multimodal data fusion according to claim 6, characterized in that, The step of calculating the initial edge weight of the directed edge based on the time cross-information value and the p-value of the Granger test includes: The maximum value among all the aforementioned time-cross mutual information values is defined as the maximum cross mutual information value. The initial edge weights are obtained by dividing the time cross mutual information value by the maximum cross mutual information value and then multiplying it by the difference between the preset value and the p value of the Granger test.
8. A dual-disease auxiliary diagnostic system based on multimodal data fusion, characterized in that, The system includes: The acquisition unit is used to construct a fusion feature matrix based on the patient's imaging data, laboratory indicators, and medical history fields, and extract a first indicator sub-matrix and a second indicator sub-matrix from the fusion feature matrix respectively; wherein, the second indicator sub-matrix includes multiple time-series variables; The scoring unit is used to score according to the first index submatrix to obtain the immune score, define each of the time series variables as a graph node, and calculate the time cross-information value and the p-value of the Granger test between each of the graph nodes. A unit is established to establish directed edges between the corresponding graph nodes if the time cross-information value is greater than a preset information value and the p value of the Granger test is less than a preset p value. The calculation unit is used to calculate the target edge weight of the directed edge based on the time cross-information value, the p-value of the Granger test, the immune score, and the preset path adjustment coefficient matrix; wherein, a mechanism causal graph is constructed based on the graph nodes and the target edge weight; The identification unit is used to input the causal graph of the mechanism into a preset graph neural network to obtain a diagnostic graph vector, and input the diagnostic graph vector into a preset fully connected neural network to obtain a main symptom identification label.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the dual-disease auxiliary diagnosis method based on multimodal data fusion as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the dual-disease auxiliary diagnosis method based on multimodal data fusion as described in any one of claims 1 to 7.