Clinical auxiliary decision method and system based on integrated time-series multi-modal data

Abstract

The application discloses a clinical auxiliary decision-making method and system based on integrated time-series multi-modal data, relates to the field of medical information technology, and comprises the following steps: collecting and time-aligning time-series multi-modal clinical data of a patient, performing feature analysis and cross-modal fusion, and generating a unified multi-dimensional time-series health state feature spectrum.Based on the feature spectrum and a medical knowledge graph, a clinical state evolution model is established, the evolution distance between the current state of the patient and each key clinical event node is calculated, and then a dynamic risk assessment surface is constructed to identify a high-risk evolution path.Aiming at the high-risk path, the key feature combination is located back, corresponding intervention measure evidence chains are searched from the knowledge graph, and finally a clinical decision support report is generated.The application realizes dynamic quantitative risk assessment of disease evolution and automatic evidence-based decision recommendation.

Description

Technical Field

[0001] This invention belongs to the field of medical information technology, specifically a clinical decision support method and system based on integrated time-series multimodal data. Background Technology

[0002] In clinical practice, patients' health status continuously generates temporal, multimodal data through monitoring equipment, laboratory systems, and imaging equipment. Current technologies typically employ data fusion and machine learning methods to integrate and analyze this multimodal data to predict disease risk or aid in diagnosis. However, these methods often focus on static risk assessment at a single point in time or within a short window, or only predict a specific clinical endpoint. They struggle to continuously and quantitatively characterize the dynamic evolution and real-time risk of a patient's health status towards multiple potential key clinical events throughout the entire course of their illness.

[0003] Existing clinical decision support systems often stop at issuing warnings after identifying high-risk areas. These systems lack a mechanism to automatically link identified high-risk pathways to specific, interpretable combinations of temporal physiological characteristics leading to the risk. Furthermore, they lack the ability to automatically match and retrieve evidence of appropriate interventions from structured medical knowledge based on these key characteristics. This results in a gap between risk warnings and specific, actionable clinical decision recommendations, leading to insufficient automation and interpretability in the decision support process. Summary of the Invention

[0004] This invention aims to solve at least one of the technical problems existing in the prior art;

[0005] Therefore, this invention proposes a clinical decision support method based on integrated temporal multimodal data, including:

[0006] Collect time-series multimodal clinical data from patients from different medical devices and information systems;

[0007] The time-series multimodal clinical data are time-aligned and missing values ​​are imputed to construct a synchronized multimodal data stream based on a unified time axis;

[0008] For each modality of data in the synchronized multimodal data stream, feature parsing is performed to extract a feature set reflecting the physiological state of the disease;

[0009] The feature sets extracted from different modalities are fused across modalities to generate a unified multi-dimensional temporal health status feature spectrum for patients.

[0010] Based on the multi-dimensional temporal health status feature spectrum, combined with a pre-set medical knowledge graph, a clinical state evolution model representing the dynamic evolution of the patient's disease course is established.

[0011] Key clinical event nodes are defined in the clinical state evolution model, and the evolution distance between the patient's current state and each key clinical event node is calculated.

[0012] Based on the evolution distance and the preset evolution threshold, a dynamic risk assessment surface is constructed to show the patient's evolution toward different key clinical events;

[0013] For the high-risk evolution path identified in the dynamic risk assessment surface, the multi-dimensional time-series health status feature spectrum is traced back to locate key feature combinations and, based on this, the evidence chain of intervention measures is retrieved from the medical knowledge graph to generate a clinical decision support report.

[0014] Furthermore, the collection of patient time-series multimodal clinical data from different medical devices and information systems specifically includes:

[0015] The time-series multimodal clinical data includes at least vital sign waveforms, medical image sequences, and structured text records;

[0016] Vital signs waveform data are continuously collected from the monitoring device interface, including electrocardiogram, blood oxygen saturation waveform, and arterial blood pressure waveform;

[0017] The patient's medical image sequence is retrieved from the image archiving and communication system in chronological order of examination time. The medical image sequence includes computed tomography (CT) scan sequence and magnetic resonance imaging (MRI) sequence.

[0018] Structured text records corresponding to patients are extracted from the hospital information system and laboratory information system. These structured text records include medical orders, laboratory test reports, and nursing records.

[0019] Each piece of the aforementioned time-series multimodal clinical data is labeled with its original acquisition time point and data source modality type.

[0020] Furthermore, the step of performing time alignment and missing value imputation on the temporal multimodal clinical data to construct a synchronized multimodal data stream based on a unified time axis specifically involves:

[0021] The timestamps of all data in the aforementioned time-series multimodal clinical data are converted to the same standard time coordinate system;

[0022] For the converted time-series multimodal clinical data, a unified time axis index is established based on a preset time granularity;

[0023] Check the integrity of each modal data at each time point in the time axis index;

[0024] For time points where data is missing, numerical interpolation is performed using interpolation based on the same modal data of adjacent time points or extrapolation based on the correlation of cross-modal data.

[0025] The interpolated modal data are reorganized according to the unified time axis index to generate a synchronized multimodal data stream with one time point corresponding to each other.

[0026] Furthermore, the step of performing feature parsing on each modality of data in the synchronized multimodal data stream to extract a feature set reflecting the physiological state of the disease specifically involves:

[0027] Time-domain, frequency-domain, and time-frequency-domain analyses are performed on the vital sign waveform data in the synchronized multimodal data stream to extract waveform morphology features, rhythm features, and energy distribution features.

[0028] The medical image sequence in the synchronized multimodal data stream is segmented and quantized based on deep learning, and radiomics features, including texture features, shape features and intensity statistical features, are extracted.

[0029] Medical entity recognition and relation extraction are performed on the structured text records in the synchronized multimodal data stream to construct clinical event features, including diagnostic events, medication events, and symptom events.

[0030] The waveform morphology features, rhythm features, energy distribution features, radiomics features, and clinical event features extracted for each modality of data are summarized into feature sets for each corresponding modality.

[0031] Furthermore, the step of fusing the feature sets extracted from different modalities across modalities to generate a unified multi-dimensional temporal health status feature spectrum for the patient specifically involves:

[0032] The feature sets of different modalities are mapped to the same high-dimensional latent feature space;

[0033] In the high-dimensional latent feature space, the correlation weights between different modal features are calculated;

[0034] Based on the aforementioned correlation weights, features from different modalities but reflecting the same physiological or pathological dimension are weighted and fused to form fused dimensional features.

[0035] All the fused dimensional features are arranged and combined according to the unified time axis index;

[0036] The unified multidimensional temporal health status feature spectrum of the patient is composed of all fused dimensional features at each time point.

[0037] Furthermore, the establishment of a clinical state evolution model representing the dynamic evolution of the patient's disease course based on the multi-dimensional temporal health state feature spectrum and combined with a preset medical knowledge graph specifically includes:

[0038] The feature vector of each time point in the multi-dimensional temporal health status feature spectrum is mapped to a state node in the clinical state evolution model.

[0039] Based on the disease development path and state transition relationship defined in the medical knowledge graph, directed connection edges are established between state nodes at adjacent time points;

[0040] Each directed connection edge is assigned a weight, which is calculated based on the magnitude of feature vector change and typical state transition patterns in the medical knowledge graph;

[0041] All state nodes and the weighted directed edges between them together constitute a clinical state evolution model describing the individualized course of a patient's disease.

[0042] Furthermore, the step of defining key clinical event nodes in the clinical state evolution model and calculating the evolution distance between the patient's current state and each key clinical event node specifically involves:

[0043] Predefined key clinical event nodes with significant clinical importance are extracted from the medical knowledge graph. These key clinical event nodes include turning points in disease deterioration, points where complications occur, and points for evaluating treatment response.

[0044] In the clinical state evolution model, the state node that is closest to the key clinical event node in the feature space is identified;

[0045] The current state node is the state node corresponding to the patient's latest time point.

[0046] Calculate the shortest path length from the current state node, along the directed connection edges in the clinical state evolution model, to the state node corresponding to each key clinical event node;

[0047] The shortest path length is the distance between the patient's current state and each key clinical event node.

[0048] Furthermore, the step of constructing a dynamic risk assessment surface for the patient's evolution towards different key clinical events based on the evolution distance and a preset evolution threshold specifically involves:

[0049] For each key clinical event node, a corresponding evolution threshold is set;

[0050] Divide the distance between the patient’s current state and each key clinical event node by its corresponding evolution threshold to obtain the normalized relative evolution urgency.

[0051] A multidimensional risk vector is constructed using different key clinical events as dimensions and relative urgency as a numerical value.

[0052] The multidimensional risk vector is mapped to a continuous two-dimensional or three-dimensional space by a nonlinear function, forming a dynamic risk assessment surface.

[0053] Each point on the dynamic risk assessment surface represents a specific combination of risk states, and its height or color indicates the overall risk level.

[0054] Furthermore, for the high-risk evolution path identified in the dynamic risk assessment surface, the multi-dimensional time-series health status feature spectrum is traced back to locate key feature combinations and, based on this, the evidence chain of intervention measures is retrieved from the medical knowledge graph to generate a clinical decision support report, including:

[0055] For the high-risk evolution path identified in the dynamic risk assessment surface, the multi-dimensional time-series health status feature spectrum is traced back to locate the key feature combination that leads to increased risk.

[0056] Based on the combination of key features, retrieve the associated chain of evidence for intervention measures from the medical knowledge graph;

[0057] By integrating the dynamic risk assessment surface, high-risk evolution path, and the evidence chain of associated intervention measures, a structured clinical decision support report is generated.

[0058] The step of retrieving the associated evidence chain of intervention measures from the medical knowledge graph based on the combination of key features specifically involves:

[0059] The key features are combined and used as query conditions, which are then input into the medical knowledge graph.

[0060] In the medical knowledge graph, match the pathophysiological state nodes that have a direct or indirect causal or correlational relationship with the combination of the key features;

[0061] Retrieve the knowledge paths that connect the pathophysiological state nodes with various clinical intervention nodes;

[0062] All retrieved knowledge paths were ranked by evidence level and scored by relevance. The top-scoring knowledge paths were selected as the evidence chain for recommended intervention measures.

[0063] The chain of evidence for intervention measures describes the logical reasoning chain from the identified key features to the specific intervention measures.

[0064] Furthermore, the present invention also includes a clinical decision support system based on integrated temporal multimodal data, the system including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements the steps of the clinical decision support method based on integrated temporal multimodal data as described above.

[0065] Compared with the prior art, the beneficial effects of the present invention are:

[0066] By calculating the evolutionary distance between a patient's current multidimensional temporal health status feature spectrum and predefined key clinical event nodes, and constructing a dynamic risk assessment surface based on evolutionary thresholds, this technology integrates discrete temporal features into a continuous quantitative measure representing state evolution, mapping the patient's complex and dynamically changing clinical state into a computable evolutionary space. The dynamic risk assessment surface can simultaneously assess the immediate probability and trajectory of a patient's state progressing towards multiple different clinical outcomes. This model overcomes the limitations of traditional static risk scoring models in the time dimension, providing a continuously updated risk profile assessment over time. Clinicians can use this surface to observe the continuous evolution trend of the patient's state, anticipate potential disease progression, and make predictions within critical time windows.

[0067] Once a high-risk progression path is identified, the system reverse-engineers it to generate a multi-dimensional temporal health status feature spectrum for that path. By analyzing the temporal correlation, magnitude of change, and patterns among the features, the system identifies the core feature combination driving the risk progression. Using this feature combination as a precise query condition, the system automatically retrieves interventions, clinical guidelines, and research evidence with strong medical logical connections from a structured medical knowledge graph, forming a complete evidence chain. Based on this, a clinical report containing decision-making basis and recommendations is generated. This process automates the process from risk identification to the generation of precise decision-making recommendations. It directly links the high-risk warnings output by the system to specific, explainable physiological abnormalities that lead to the risk, and bases decision-making recommendations on structured medical evidence. This chain bridges the gap between risk warnings and clinical action plans, reducing the time and cognitive load required for doctors to manually search for and match evidence, and improving the efficiency and evidence-based reliability of the decision-making process. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating the steps of the clinical decision support method based on integrated temporal multimodal data described in this invention.

[0069] Figure 2 A flowchart for time alignment and missing value interpolation;

[0070] Figure 3A flowchart for feature parsing of multimodal data;

[0071] Figure 4 The normalized Euclidean distance thermodynamic matrix for state node features;

[0072] Figure 5 A graph showing the correlation between evidence scores and knowledge path jumps for different acute kidney injury interventions. Detailed Implementation

[0073] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] See Figure 1 The system collects temporal multimodal clinical data from patients using different medical devices and information systems; performs time alignment and missing value imputation on the temporal multimodal clinical data to construct a synchronized multimodal data stream based on a unified time axis; performs feature analysis on each modality of the synchronized multimodal data stream to extract feature sets reflecting the physiological state of the disease; fuses the feature sets extracted from different modalities across modalities to generate a unified multidimensional temporal health status feature spectrum for patients; and establishes a system based on the multidimensional temporal health status feature spectrum and a pre-set medical knowledge graph. A clinical state evolution model characterizing the dynamic evolution of a patient's disease course is used. Key clinical event nodes are defined in the clinical state evolution model, and the evolution distance between the patient's current state and each key clinical event node is calculated. Based on the evolution distance and a preset evolution threshold, a dynamic risk assessment surface is constructed to represent the patient's evolution towards different key clinical events. For high-risk evolution paths identified in the dynamic risk assessment surface, the multi-dimensional temporal health state feature spectrum is traced back to locate key feature combinations and, based on this, the evidence chain of intervention measures is retrieved from the medical knowledge graph to generate a clinical decision support report.

[0075] In one embodiment of the present invention, the acquisition of patient time-series multimodal clinical data from different medical devices and information systems is a process involving the integration and annotation of heterogeneous data sources. The time-series multimodal clinical data includes at least vital sign waveforms, medical image sequences, and structured text records. Taking a patient suspected of septic shock in an intensive care unit (ICU) as an example, the acquisition of vital sign waveform data is achieved by connecting to bedside monitoring equipment through a programming interface. Vital sign waveform data is continuously acquired from the monitoring equipment interface. The electrocardiogram is acquired at a frequency of 125 Hz and includes voltage signals from twelve leads. The blood oxygen saturation waveform reflects the light absorption changes of pulsatile vessels, and the arterial blood pressure waveform is the pressure-pulse curve measured through an arterial catheter. These waveform data are acquired in real time with a millisecond-level time resolution and the original timestamps of the data points generated by the device are appended. The acquisition of medical imaging sequences is accomplished through the hospital's internal image archiving and communication system. Patient medical imaging sequences are retrieved from the system in chronological order of examination, such as chest computed tomography (CT) scans performed on the day of admission, the third day, and the seventh day. Each CT scan sequence contains dozens of axial slice images from the base to the top of the lung. Magnetic resonance imaging (MRI) sequences may include weighted imaging sequences used to assess tissue perfusion. Each sequence file embeds a standard medical digital imaging and communication protocol header file, containing precise examination date and time information. The acquisition of structured text records is achieved by querying the database interfaces of the hospital information system and the laboratory information system. Structured text records corresponding to the patient are extracted from these systems. The hospital information system provides medical orders containing drug names, dosages, routes of administration, and prescribing times. The laboratory information system provides laboratory test reports containing numerical results such as white blood cell count, procalcitonin, and lactate levels, along with sample collection times. Nursing records contain regularly entered vital sign measurements, consciousness scores, and fluid intake and output data. All these records are stored in a relational database in structured field format.

[0076] In some embodiments, labeling each piece of time-series multimodal clinical data with its original acquisition time and data source modality is a necessary step to ensure data traceability. For vital sign waveform data continuously acquired from the monitoring device interface, each data packet is marked with an acquisition time accurate to milliseconds, synchronized with the device clock, and labeled with the data source modality as "continuous vital sign waveform". For medical image sequences retrieved from the image archiving and communication system in chronological order of examination, the "study date and time" field in the medical digital imaging and communication protocol header file of each medical image sequence is extracted as the acquisition time and labeled with the data source modality as "medical image sequence". For structured text records extracted from hospital information systems and laboratory information systems, the "record creation time" or "sample collection time" field associated with each record is extracted as the acquisition time. The data source modality of medical order records is labeled as "structured medical order text", the data source modality of laboratory test reports is labeled as "structured test text", and the data source modality of nursing records is labeled as "structured nursing text". In this way, all the collected raw data have a unified time reference and modal identifier.

[0077] It is understandable that there are inherent differences in the acquisition frequency and time granularity of different modalities of data. The vital sign waveform data continuously acquired from the monitoring device interface is a near-continuous time series, while the medical image sequence retrieved from the image archiving and communication system according to the examination time sequence is a discrete time point event. Furthermore, the structured text records extracted from the hospital information system and laboratory information system may have a discrete or uneven time distribution. This difference in temporal characteristics requires special handling in subsequent steps. In a specific comparative example, for the same patient between 14:00 and 14:30 on October 26, 2023, the vital sign waveform data continuously acquired from the monitoring device interface provided approximately 7500 ECG sampling points, blood oxygen saturation waveforms, and arterial blood pressure waveforms per minute. However, the structured text records extracted from the hospital information system and laboratory information system may only contain a laboratory test report acquired at 14:15 and several nursing record entries spaced several minutes apart. The medical image sequence retrieved from the image archiving and communication system according to the examination time sequence may have no data during this time period. This contrast between data density and periodicity highlights the asynchronicity and sparsity of multimodal data over time.

[0078] See Figure 2In one embodiment of the present invention, time alignment and missing value imputation of temporal multimodal clinical data are key steps in constructing a synchronized multimodal data stream. Taking the monitoring of a sepsis patient within 48 hours of admission as an example, the temporal multimodal clinical data includes vital sign waveform data continuously collected from the monitoring device interface, a chest computed tomography sequence retrieved from the image archiving and communication system, and structured text records extracted multiple times from the hospital information system and laboratory information system. The primary operation is to convert the timestamps of all data to the same standard time coordinate system. The device local time recorded in the electrocardiogram waveform data packet, the examination time in the medical digital imaging and communication protocol header file of the computed tomography sequence, and the acquisition time recorded in the laboratory test report database are all converted to the format "2023-10-27 10:00:00.000" based on Coordinated Universal Time (UTC), eliminating potential time zone differences or clock deviations between different medical devices and information systems.

[0079] For the converted time-series multimodal clinical data, a unified time axis index is established based on a preset time granularity. In the intensive care unit scenario, the time granularity can be set to one minute. Accordingly, a continuous time point sequence indexed in minutes is established for the 48 hours starting from the patient's admission time "2023-10-27 10:00:00", i.e., T1 (10:00), T2 (10:01)... up to T2880 (10:00 the next day). The integrity of each modality data at each time point in the time axis index is checked. For example, at the time point "2023-10-27 15:30:00", the system will check for the existence of vital sign waveform data, medical image sequence data, and structured text record data. Because computed tomography (CT) scan sequences are discrete events, medical image sequence data will be missing at most points in time; while vital sign waveform data should theoretically exist at every point in time, but may be missing due to momentary signal interruption; structured text records such as nursing entries may appear at specific points in time.

[0080] For time points with missing data, numerical interpolation is performed using interpolation based on adjacent time points with the same modality or extrapolation based on the correlation of cross-modal data. In practice, if the heart rate value is missing at a certain time point, but the heart rate values ​​for the previous and next minutes are available, linear interpolation is used to calculate the missing value. The interpolation process can be expressed by the following formula:

[0081]

[0082] in: This represents the heart rate value that needs to be interpolated at time point t. Indicates the previous valid time point Heart rate value, Indicates the next valid time point Heart rate values. For modalities such as medical image sequences that are completely missing over long time intervals, extrapolation based on cross-modal data correlation may be enabled. For example, when image data is missing for a certain time period in the timeline index, but the vital signs waveform features and inflammatory indicators in the structured text records show high abnormalities during that time period, the system can generate a quantitative feature estimate indicating "suspected worsening of image abnormalities" for interpolation based on the cross-modal correlation model established in historical training data, rather than directly generating pixel-level image data.

[0083] In some embodiments, the choice of time granularity can be adjusted according to clinical monitoring needs. For high-frequency physiological signal analysis, the time granularity can be set to the second or even millisecond level. At this time, the density of the time axis index increases significantly, placing higher real-time requirements on the data integrity checking and interpolation algorithms. It can be understood that the preset time granularity determines the temporal resolution of the synchronized multimodal data stream. Finer granularity can capture faster pathophysiological changes, but it also increases the computational complexity and the requirements for data interpolation reliability.

[0084] After interpolation, the modal data are reassembled according to a unified timeline index to generate a synchronized multimodal data stream with one-to-one correspondence between time points. After reassembly, each time point index corresponds to a structured data packet. For example, the data packet corresponding to the time point "2023-10-27 22:05:00" contains the complete interpolated ECG waveform segment, blood oxygen saturation waveform segment, and arterial blood pressure waveform segment for that minute. It also marks the image feature estimates that may be extrapolated but for which there is no original medical image sequence data at this time, as well as all structured text records recorded at this time point, such as a medical order to "administer fluid resuscitation" and a nursing record stating that "central venous pressure measurement is 8 mmHg." Through this reassembly, the originally asynchronous and density-heterogeneous original multimodal data is transformed into a regular data table or data stream that progresses according to a unified time step.

[0085] See Figure 3In one embodiment of the present invention, feature parsing is performed on each modality of data in the synchronized multimodal data stream to extract feature sets reflecting the physiological state of the disease, and the feature sets extracted from different modalities are fused across modalities to generate a unified multidimensional temporal health status feature spectrum for the patient. This constitutes the core transformation process from raw data to integrated feature representation. This process performs time-domain, frequency-domain, and time-frequency-domain analysis on the vital sign waveform data in the synchronized multimodal data stream to extract waveform morphology features, rhythm features, and energy distribution features. For example, for electrocardiogram waveform segments, time-domain analysis can calculate waveform morphology features such as RR interval, QRS width, and ST segment elevation; frequency-domain analysis extracts rhythm features such as high-frequency power and low-frequency power in heart rate variability through fast Fourier transform; and time-frequency-domain analysis characterizes the energy of the electrocardiogram signal in time and frequency through wavelet transform. Joint distribution features in the frequency dimension; for medical image sequences in synchronized multimodal data streams, deep learning-based lesion region segmentation and quantization are performed to extract radiomics features, including texture features, shape features, and intensity statistics features. For example, for a single time point image of a chest computed tomography sequence, a pre-trained U-Net network is used to segment the lung infection region, and then texture features such as contrast and correlation of the gray-level co-occurrence matrix are calculated from the segmented region, shape features such as lesion volume and sphericity are calculated, and intensity statistics features such as mean and skewness of pixel values ​​within the region are calculated. For structured text records in synchronized multimodal data streams, medical entity recognition and relation extraction are performed to construct clinical event features, including diagnostic events, medication events, and symptom events, thereby constructing a structured "vasoactive drug use" medication event feature. The waveform morphology features, rhythm features, energy distribution features, radiomics features, and clinical event features extracted for each modality of data are summarized into feature sets for each corresponding modality. For example, all numerical features extracted from electrocardiograms, blood oxygen saturation waveforms, and arterial blood pressure waveforms are merged into a vital signs modality feature set; all quantitative features extracted from computed tomography sequences and magnetic resonance imaging sequences are merged into a medical imaging modality feature set; and all event vectors parsed from medical orders, laboratory test reports, and nursing records are merged into a text record modality feature set.

[0086] In some embodiments, cross-modal fusion of feature sets from different modalities begins by mapping these feature sets to the same high-dimensional latent feature space. This can be achieved by training a fully connected neural network encoder for each modal feature set, where each encoder transforms the original feature vector of its corresponding modality into a latent feature vector with the same dimension. Calculating the correlation weights between features from different modalities in the high-dimensional latent feature space is crucial for fusion. One implementation employs an attention-based weight calculation method. For feature vectors from different modalities that may reflect the same physiological dimension, their interaction attention scores are calculated to determine their contribution during fusion. The calculation of correlation weights can be expressed as the following formula:

[0087]

[0088] in: Indicates the first A certain feature of the modality (as a query) and the first modality Normalized correlation weights between a certain feature (as a key) of a modality. It is the first Query vector projection of each modal feature It is the first Key vector projection of each modal feature Represents a vector similarity function (such as a dot product). It is the dimension of the key vector. This represents the total number of features involved in the calculation. Based on relevance weights, features from different modalities that reflect the same physiological or pathological dimension are weighted and fused to form fused dimensional features. For example, a dimension reflecting the risk of "circulatory failure" might be formed by weighted fusion of hypotension features from the vital signs modality, cardiac ejection fraction features from the medical imaging modality, and vasoactive drug use features from the text recording modality. All fused dimensional features are arranged and combined according to a unified timeline index. The patient's unified multi-dimensional temporal health status feature spectrum consists of all fused dimensional features at each time point, ultimately forming a three-dimensional data structure. Its three dimensions are the time point, the fused physiological / pathological dimension, and the quantified feature value of that dimension.

[0089] In one embodiment of the present invention, a clinical state evolution model representing the dynamic evolution of a patient's disease course is established based on a multi-dimensional temporal health state feature spectrum combined with a preset medical knowledge graph. Key clinical event nodes are defined in the clinical state evolution model, and the evolution distance between the patient's current state and each key clinical event node is calculated. Based on the evolution distance and a preset evolution threshold, a dynamic risk assessment surface is constructed to represent the patient's evolution towards different key clinical events. This series of steps realizes the transformation from static features to dynamic risk quantification assessment. Specifically, the feature vector of each time point in the multi-dimensional temporal health state feature spectrum is... This is mapped to a state node in the clinical state evolution model. For example, for a patient with heart failure, the feature vector at time point T1 (at admission) may contain a high NT-proBNP value, a normal ejection fraction, and a mild dyspnea score. After normalization, this feature vector constitutes a state node named "S1" in the clinical state evolution model. At time point T2 (6 hours after admission), the feature vector may evolve into a further increase in the NT-proBNP value, a slight decrease in the ejection fraction, and a moderate dyspnea score. This feature vector constitutes a state node named "S2". Based on the disease progression paths and state transition relationships defined in the medical knowledge graph, directed connections are established between state nodes at adjacent time points. The medical knowledge graph may contain state transition relationships such as "increased volume overload" potentially leading to "decreased cardiac output," and "decreased cardiac output" potentially leading to "insufficient end-organ perfusion." If the direction of change of the feature vector from S1 to S2 conforms to the typical feature change pattern of the transition from "compensated heart failure" to "decompensated heart failure" in the medical knowledge graph, then a directed connection is established between state nodes S1 and S2. Each directed connection is assigned a weight, calculated based on the magnitude of the feature vector change and the typical state transition pattern in the medical knowledge graph. The formula for calculating the weight can be designed as follows:

[0090]

[0091] in: Indicates from the state node To the state node The weights of the directed connection edges. It is a state node eigenvectors With state nodes eigenvectors The Euclidean distance between them is used to quantify the magnitude of change in the feature vector. It is a matching degree function used to measure the amount of change in feature vectors. With medical knowledge graph The degree of matching with predefined typical state transition patterns; the higher the degree of matching, the larger the function value. It is a harmonic coefficient between 0 and 1, used to balance the magnitude of feature vector changes with the weighting of prior knowledge from the medical knowledge graph in the weight calculation. All state nodes and the weighted directed edges between them together constitute a clinical state evolution model describing the individualized course of the patient's disease. Structurally, this model is a weighted directed graph, and the sequence of nodes in the graph depicts the actual state evolution trajectory of the patient from admission to the current time.

[0092] Predefined key clinical event nodes with significant clinical importance are extracted from a medical knowledge graph. These nodes include turning points in disease progression, complication occurrences, and treatment response assessment points. For example, in a medical knowledge graph related to heart failure, "acute pulmonary edema," "cardiogenic shock," and "acute kidney injury" might be predefined as turning points in disease progression or complication occurrences, while "good response to diuretic therapy" might be a treatment response assessment point. In the clinical state evolution model, the state node closest to the key clinical event node in the feature space is identified. This involves calculating the similarity between the feature vector of each state node in the clinical state evolution model and the standard feature vector of each key clinical event node, and marking the state node with the highest similarity as its corresponding point in the model. Using the state node corresponding to the patient's latest time point as the current state node, the shortest path length from the current state node, along the directed edges in the clinical state evolution model, to the state node corresponding to each key clinical event node is calculated. The shortest path length can be calculated using Dijkstra's algorithm, accumulating the weights of all directed edges along the path. The shortest path length is the distance between the patient's current state and each key clinical event node. The magnitude of the distance reflects the theoretical difficulty or "cost" required to evolve from the patient's current and historical state to a certain key clinical event. See Table 1 for an example calculation result.

[0093] Table 1: Key Clinical Event Milestones and Evolution Distance

[0094]

[0095] For each critical clinical event node, a corresponding evolution threshold is set. This threshold can be statistically derived from historical population data; for example, the evolution threshold for the "acute pulmonary edema" event might be set to 20. The evolution distance between the patient's current state and each critical clinical event node is divided by its corresponding evolution threshold to obtain a normalized relative evolution urgency. A relative evolution urgency less than 1 indicates that if the current evolution rate continues, the event node may be reached within the threshold time. Using different critical clinical events as dimensions and relative evolution urgency as values, a multidimensional risk vector is constructed. For example, for the four critical clinical events mentioned above, a four-dimensional risk vector can be constructed. A dynamic risk assessment surface is formed by mapping a multidimensional risk vector to a continuous two- or three-dimensional space using a nonlinear function. One mapping method involves reducing the four-dimensional risk vector to two dimensions using principal component analysis, and then generating a continuous surface using kernel density estimation. Each point on the dynamic risk assessment surface represents a combination of risk states, and its height or color indicates the overall risk level.

[0096] See Figure 4 This study presents pairwise similarity measures for six patient health status nodes (S0 to S5) in a high-dimensional feature space. The values ​​and corresponding thermochromatograms, symmetrically distributed in the upper and lower triangular sections of the matrix, accurately characterize the normalized Euclidean distance between the starting and target state nodes: values ​​closer to 0 (bright yellow on the chromatogram) indicate smaller differences in the multi-dimensional temporal health feature spectra of the two nodes, and closer their pathophysiological states (e.g., S4 and S5 have a distance of 0.32, and S2 and S3 have a distance of 0.38); values ​​closer to 2.0 (deep wine red on the chromatogram) indicate significant state differences and a larger span in the evolution of pathophysiological states (e.g., S0 and S5 have a distance of 2.0, and S0 and S4 have a distance of 1.68). In the construction process of the clinical state evolution model, the values ​​of this matrix directly provide core parameters for calculating the weights of directed connections, representing the amplitude of feature vector changes. Specifically, each off-diagonal element in the matrix can be used as a weight in the formula. The quantized value is combined with the predefined state transition matching degree function in the medical knowledge graph. With harmonic coefficient This allows for the assignment of weights to the transition edges between state nodes. The normalized distance matrix also provides a similarity criterion for matching key clinical event nodes, which can help identify the in-model state node that is closest to the feature vector of a preset key clinical event (such as a turning point in disease deterioration). This lays a standardized feature difference measurement foundation for subsequent evolution distance calculation and dynamic risk assessment surface construction.

[0097] In one embodiment of the present invention, a multi-dimensional time-series health status feature spectrum is backtracked for high-risk evolution paths identified in the dynamic risk assessment surface to locate key feature combinations. Based on this, an evidence chain of intervention measures is retrieved from the medical knowledge graph to ultimately generate a clinical decision support report. This process completes a closed loop from risk warning to clinical decision recommendations. The high-risk evolution path identified in the dynamic risk assessment surface may manifest as a path pointing to the key clinical event node of "acute kidney injury," with its relative urgency jumping from 0.5 to 0.9 in a short period of time. The system then initiates a backtracking analysis procedure to backtrack multiple... The dimensional time-series health status feature spectrum is used to locate the key feature combination that leads to increased risk. Retrospective analysis examines the fused dimensional feature values ​​corresponding to the recent state nodes that constitute the high-risk path along the time axis. By comparing the differences in the feature spectrum before and after the risk jump, the study locates the following: a continuous decrease in the "urine volume" feature value and a sharp increase in the "serum creatinine" feature value in the "renal function" dimension; a slow decrease in the "mean arterial pressure" feature value in the "hemodynamics" dimension; and a high "white blood cell count" feature value in the "inflammation" dimension. This set of synergistic abnormal changes in feature values ​​is identified as the key feature combination.

[0098] Based on key feature combinations, evidence chains of related interventions are retrieved from the medical knowledge graph. These key feature combinations are input as query conditions into the medical knowledge graph, where they are parsed into a series of entities and relational assertions, such as "decreased urine output," "elevated serum creatinine," "decreased mean arterial pressure," and "elevated white blood cell count." Pathophysiological state nodes with direct or indirect causal or correlational relationships to the key feature combinations are matched within the medical knowledge graph. For example, "decreased urine output" and "elevated serum creatinine" might both point to the "acute kidney injury" state node, "decreased mean arterial pressure" might be associated with the "inadequate renal perfusion" state node, and "elevated white blood cell count" might be associated with the "systemic inflammatory response" state node. Knowledge paths connecting pathophysiological state nodes and various clinical intervention nodes are retrieved. Starting from the "acute kidney injury" and "inadequate renal perfusion" state nodes, the medical knowledge graph might contain intervention nodes such as "optimize volume status," "avoid nephrotoxic drugs," and "consider renal replacement therapy," along with relational edges connecting these states and interventions such as "recommended for" and "may be beneficial." All retrieved knowledge paths are ranked by evidence level and scored by relevance. The top-scoring knowledge paths are selected as the recommended intervention evidence chains. The evidence level ranking is based on the source of medical evidence involved in the path, such as randomized controlled trials, cohort studies, or expert consensus, and different weights are assigned accordingly. The relevance score comprehensively considers the semantic similarity between query features and nodes in the path, as well as the path length. The intervention evidence chain describes the logical reasoning chain from the identified key features to the specific intervention. A structured clinical decision support report is generated by integrating the dynamic risk assessment surface, high-risk evolution paths, and associated intervention evidence chains. The report is presented in a standardized format and includes patient identification, risk warning summary, key feature combinations and their time-series changes identified through retrospective analysis, the recommended intervention evidence chain, and an overview of its supporting evidence.

[0099] In some embodiments, the specific calculation of evidence level ranking and relevance score for all retrieved knowledge paths can employ a combination of graph-based ranking algorithms and rule-weighted methods. The formula for calculating the relevance score can be defined as follows:

[0100]

[0101] in: Represents a knowledge path The final score, It is a path The set of all relation edges traversed. It is based on the relation edge The pre-set weights for the associated evidence levels (e.g., Level I, Level II, Level III evidence), It is an indicator function that takes the value 0 or 1, when the relation edge... The value is 1 if the type is highly relevant to the context of the query feature combination, otherwise it is 0. It is a path The number of hops (i.e., the number of nodes traversed minus one). This is a moderating parameter used to adjust for the impact of path length. The formula encourages the selection of knowledge paths that are concise and involve strong correlations with high levels of evidence.

[0102] It is understandable that the location of key feature combinations is not limited to feature values ​​at a single point in time, but focuses more on the changing trends and patterns of features within the risk escalation time window. The completeness and quality of the medical knowledge graph directly determine the clinical rationality and practicality of the retrieved intervention evidence chains. Structured clinical decision support reports do not replace clinicians' judgments, but rather provide integrated information that has been traced and reasoned to assist in decision-making. Optionally, when generating clinical decision support reports, a visual screenshot of the dynamic risk assessment surface and a schematic diagram of the position of high-risk evolution paths in the clinical state evolution model can be attached to enhance the interpretability of the report. When retrieving intervention evidence chains, in addition to using a predefined medical knowledge graph, online clinical guideline databases or the latest medical literature abstracts can also be accessed, and relevant intervention suggestions can be extracted in real time using natural language processing technology to supplement the evidence chains. For several selected intervention evidence chains, they can be reordered and merged according to their potential synergistic effects or mutual contraindications to form a coordinated, prioritized list of intervention recommendations.

[0103] See Figure 5The graph presents the correlation between the evidence scores of different clinical interventions and the number of hops in the corresponding knowledge path. Specifically, the graph uses a bar chart to display the evidence scores (0-1) of five interventions: "optimizing volume status," "avoiding nephrotoxic drugs," "renal replacement therapy," "anti-inflammatory therapy," and "vasopressor therapy." The scores are 0.95, 0.92, 0.88, 0.85, and 0.80, respectively. The red line represents the number of hops in the corresponding knowledge path of each intervention in the medical knowledge graph, ranging from 2.0 to 3.0. As can be seen from the graph, the interventions "optimizing volume status" and "avoiding nephrotoxic drugs" have the highest evidence scores, at 0.95 and 0.92, respectively, and the lowest number of hops in the corresponding knowledge path (approximately 2.0). This indicates that their evidence comes from high-level Level I RCT evidence, and that the knowledge path is concise with a short reasoning chain, conforming to the optimal scoring principle of "high evidence level + strong correlation + short path" in the formula. The knowledge path hop count for "renal replacement therapy" and "vasopressor therapy" rose to 3.0. Although the former's evidence score remained at a relatively high level (0.88, Level II cohort study), the increase in the number of hops reflects its longer reasoning chain and more intermediate nodes. The evidence score for "anti-inflammatory therapy" (0.85, Level II cohort study) and the number of hops (approximately 2.0) were at a moderate level. The evidence score for "vasopressor therapy" was the lowest (0.80, Level III expert consensus), but its number of hops also reached 3.0, reflecting the combination of low evidence level and long path.

[0104] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A clinical decision support method based on integrated time-series multimodal data, characterized in that, Includes the following steps: Collect time-series multimodal clinical data from patients from different medical devices and information systems; The time-series multimodal clinical data are time-aligned and missing values ​​are imputed to construct a synchronized multimodal data stream based on a unified time axis; For each modality of data in the synchronized multimodal data stream, feature parsing is performed to extract a feature set reflecting the physiological state of the disease; The feature sets extracted from different modalities are fused across modalities to generate a unified multi-dimensional temporal health status feature spectrum for patients. Based on the aforementioned multi-dimensional temporal health status feature spectrum, and combined with a pre-defined medical knowledge graph, a clinical state evolution model representing the dynamic evolution of the patient's disease course is established, specifically as follows: The feature vector of each time point in the multi-dimensional temporal health state feature spectrum is mapped to a state node in the clinical state evolution model; and directed connections are established between state nodes at adjacent time points based on the disease development path and state transition relationship defined in the medical knowledge graph. Each directed connection edge is assigned a weight, which is calculated based on the magnitude of feature vector change and typical state transition patterns in the medical knowledge graph; All state nodes and the weighted directed edges between them together constitute a clinical state evolution model describing the individualized course of a patient's disease. Key clinical event nodes are defined in the clinical state evolution model, and the evolution distance between the patient's current state and each key clinical event node is calculated. Based on the evolution distance and the preset evolution threshold, a dynamic risk assessment surface is constructed to show the patient's evolution toward different key clinical events; For the high-risk evolution paths identified in the dynamic risk assessment surface, the multi-dimensional time-series health status feature spectrum is traced back to locate key feature combinations. Based on this, the evidence chain of intervention measures is retrieved from the medical knowledge graph to generate a clinical decision support report, including: For the high-risk evolution path identified in the dynamic risk assessment surface, the multi-dimensional time-series health status feature spectrum is traced back to locate the key feature combination that leads to increased risk; based on the key feature combination, the evidence chain of related intervention measures is retrieved from the medical knowledge graph; and by integrating the dynamic risk assessment surface, the high-risk evolution path, and the evidence chain of related intervention measures, a structured clinical decision support report is generated. The step of retrieving the associated evidence chain of intervention measures from the medical knowledge graph based on the combination of key features specifically involves: The key feature combination is used as a query condition and input into the medical knowledge graph. In the medical knowledge graph, pathophysiological state nodes that have a direct or indirect causal or correlational relationship with the key feature combination are matched. Knowledge paths connecting the pathophysiological state nodes and various clinical intervention measure nodes are retrieved. All retrieved knowledge paths are ranked by evidence level and relevance score, and the top-scoring knowledge paths are selected as recommended intervention measure evidence chains. The intervention measure evidence chain describes the logical reasoning chain from the identified key features to the specific intervention measures.

2. The clinical decision support method based on integrated temporal multimodal data according to claim 1, characterized in that, The collection of patient time-series multimodal clinical data from different medical devices and information systems specifically includes: The time-series multimodal clinical data includes at least vital sign waveforms, medical image sequences, and structured text records; Vital signs waveform data are continuously collected from the monitoring device interface, including electrocardiogram, blood oxygen saturation waveform, and arterial blood pressure waveform; The patient's medical image sequence is retrieved from the image archiving and communication system in chronological order of examination time. The medical image sequence includes computed tomography (CT) scan sequence and magnetic resonance imaging (MRI) sequence. Structured text records corresponding to patients are extracted from the hospital information system and laboratory information system. These structured text records include medical orders, laboratory test reports, and nursing records. Each piece of the aforementioned time-series multimodal clinical data is labeled with its original acquisition time point and data source modality type.

3. A clinical decision support method based on integrated temporal multimodal data according to claim 2, characterized in that, The process of performing time alignment and missing value imputation on the temporal multimodal clinical data to construct a synchronized multimodal data stream based on a unified time axis specifically involves: The timestamps of all data in the aforementioned time-series multimodal clinical data are converted to the same standard time coordinate system; For the converted time-series multimodal clinical data, a unified time axis index is established based on a preset time granularity; Check the integrity of each modality data at each time point in the time axis index; For time points where data is missing, numerical interpolation is performed using interpolation based on the same modal data of adjacent time points or extrapolation based on the correlation of cross-modal data. After interpolation, the modal data are reorganized according to the unified time axis index to generate a synchronized multimodal data stream with one time point corresponding to each other.

4. A clinical decision support method based on integrated temporal multimodal data according to claim 3, characterized in that, The step of performing feature parsing on each modality of data in the synchronized multimodal data stream to extract a feature set reflecting the physiological state of the disease specifically involves: Time-domain, frequency-domain, and time-frequency-domain analyses are performed on the vital sign waveform data in the synchronized multimodal data stream to extract waveform morphology features, rhythm features, and energy distribution features. The medical image sequence in the synchronized multimodal data stream is segmented and quantized based on deep learning, and radiomics features, including texture features, shape features and intensity statistical features, are extracted. Medical entity recognition and relation extraction are performed on the structured text records in the synchronized multimodal data stream to construct clinical event features, including diagnostic events, medication events, and symptom events. The waveform morphology features, rhythm features, energy distribution features, radiomics features, and clinical event features extracted for each modality of data are summarized into feature sets for each corresponding modality.

5. A clinical decision support method based on integrated temporal multimodal data according to claim 4, characterized in that, The step of fusing the feature sets extracted from different modalities across modalities to generate a unified multi-dimensional temporal health status feature spectrum for patients specifically involves: The feature sets of different modalities are mapped to the same high-dimensional latent feature space; In the high-dimensional latent feature space, the correlation weights between different modal features are calculated; Based on the aforementioned correlation weights, features from different modalities but reflecting the same physiological or pathological dimension are weighted and fused to form fused dimensional features. All the fused dimensional features are arranged and combined according to the unified time axis index; The unified multidimensional temporal health status feature spectrum of the patient is composed of all fused dimensional features at each time point.

6. A clinical decision support method based on integrated temporal multimodal data according to claim 5, characterized in that, The process involves defining key clinical event nodes in the clinical state evolution model and calculating the evolution distance between the patient's current state and each key clinical event node, specifically as follows: Predefined key clinical event nodes with significant clinical importance are extracted from the medical knowledge graph. These key clinical event nodes include turning points in disease deterioration, points where complications occur, and points for evaluating treatment response. In the clinical state evolution model, the state node that is closest to the key clinical event node in the feature space is identified; The current state node is the state node corresponding to the patient's latest time point. Calculate the shortest path length from the current state node, along the directed connection edges in the clinical state evolution model, to the state node corresponding to each key clinical event node; The shortest path length is the distance between the patient's current state and each key clinical event node.

7. A clinical decision support method based on integrated temporal multimodal data according to claim 6, characterized in that, The process involves constructing a dynamic risk assessment surface for the patient's progression towards different key clinical events based on the evolution distance and a preset evolution threshold, specifically as follows: For each key clinical event node, a corresponding evolution threshold is set; Divide the distance between the patient’s current state and each key clinical event node by its corresponding evolution threshold to obtain the normalized relative evolution urgency. A multidimensional risk vector is constructed using different key clinical events as dimensions and relative urgency as a numerical value. The multidimensional risk vector is mapped to a continuous two-dimensional or three-dimensional space by a nonlinear function, forming a dynamic risk assessment surface. Each point on the dynamic risk assessment surface represents a specific combination of risk states, and its height or color indicates the overall risk level.

8. A clinical decision support system based on integrated time-series multimodal data, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a clinical decision support method based on integrated temporal multimodal data as described in any one of claims 1 to 7.

Images