Clinical aid decision-making method and system based on integrated time sequence multi-modal data

Abstract

The invention discloses a clinical auxiliary decision-making method and system based on integrated time sequence multi-modal data, and relates to the technical field of medical information, and the method comprises the steps: collecting time sequence multi-modal clinical data of a patient, carrying out the time alignment, carrying out the feature analysis and cross-modal fusion, and generating a unified multi-dimensional time sequence health state feature spectrum. On the basis of the characteristic spectrum and in combination with a medical knowledge graph, a clinical state evolution model is established, the evolution distance between the current state of the patient and each key clinical event node is calculated, and then a dynamic risk assessment curved surface is constructed to recognize a high-risk evolution path. And for the high-risk path, backtracking and positioning the key feature combination, retrieving a corresponding intervention measure evidence chain from the knowledge graph, and finally generating a clinical decision support report. According to the invention, dynamic quantitative risk assessment of disease course evolution and automatic evidence-based decision recommendation are realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical information technology, and in particular to a clinical auxiliary decision-making method and system based on integrated time-series multi-modal data. BACKGROUND

[0002] In clinical practice, the health status of patients is continuously generated by time-series multi-modal data from monitoring devices, laboratory systems, imaging devices, etc. The existing technology usually adopts data fusion and machine learning methods to integrate and analyze these multi-modal data to achieve disease risk prediction or auxiliary diagnosis. However, these methods mostly focus on static risk assessment at a single time point or a short-term window, or only predict a certain specific clinical endpoint event. They are difficult to continuously and quantitatively represent the trend and real-time risk of the dynamic evolution of the health status of patients to multiple potential key clinical events throughout the course of the disease.

[0003] The existing clinical auxiliary decision-making system often stops at issuing an early warning after identifying a high risk. The system lacks a mechanism to automatically associate the identified high-risk path to a specific, interpretable combination of time-series physiological features that leads to the risk. At the same time, it also lacks the ability to automatically match and retrieve corresponding intervention measures evidence from structured medical knowledge based on these key features. This results in a gap between risk warning and specific, actionable clinical decision recommendations, and insufficient automation and interpretability of the decision support process. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art; To this end, the present application proposes a clinical auxiliary decision-making method based on integrated time-series multi-modal data, comprising: Collecting time-series multi-modal clinical data of patients from different medical devices and information systems; Time aligning and missing value imputing the time-series multi-modal clinical data to construct a synchronized multi-modal data stream with a unified time axis as the reference; Respectively performing feature analysis on each modality data in the synchronized multi-modal data stream to extract a feature set reflecting the physiological state of the disease; Cross-modal fusion of the feature sets extracted by different modalities to generate a unified multi-dimensional time-series health status feature spectrum of the patient; Based on the multi-dimensional time-series health status feature spectrum, a clinical state evolution model representing the dynamic evolution of the patient's course of disease is established in combination with a pre-set medical knowledge graph; Defining key clinical event nodes in the clinical state evolution model and calculating the evolution distance of the patient's current state from each key clinical event node; According to the evolution distance and a preset evolution threshold, a dynamic risk assessment surface of evolution of the patient to different key clinical events is constructed; For a high-risk evolution path identified in the dynamic risk assessment surface, a multi-dimensional time-series health status feature spectrum is traced back to locate a key feature combination and retrieve an intervention measure evidence chain from a medical knowledge graph to generate a clinical decision support report.

[0005] Further, the patient time-series multi-modal clinical data from different medical devices and information systems are collected, specifically: The time-series multi-modal clinical data at least include vital sign waveforms, medical image sequences and structured text records; The vital sign waveform data including electrocardiogram, blood oxygen saturation waveform and arterial blood pressure waveform are continuously collected from a monitoring device interface; The medical image sequences of the patient are retrieved from a picture archiving and communication system in chronological order, and the medical image sequences include computed tomography sequences and magnetic resonance imaging sequences; The structured text records corresponding to the patient are extracted from a hospital information system and a laboratory information system, and the structured text records include medical order records, laboratory test reports and nursing records; Each of the time-series multi-modal clinical data is labeled with an original collection time point and a data source modal type.

[0006] Further, the time-series multi-modal clinical data are time-aligned and missing value interpolated to construct a synchronized multi-modal data stream with a unified time axis as a reference, specifically: The time stamps of all data in the time-series multi-modal clinical data are converted to a same standard time coordinate system; A unified time axis index is established based on a preset time granularity for the converted time-series multi-modal clinical data; The integrity of each modal data at each time point of the time axis index is checked; For the time points with missing data, an interpolation method based on adjacent time point same modal data or an extrapolation method based on cross-modal data correlation is used for numerical interpolation; The interpolated modal data are reorganized according to the unified time axis index to generate a synchronized multi-modal data stream with time points corresponding to each other.

[0007] Further, each modal data in the synchronized multi-modal data stream is respectively feature-analyzed to extract a feature set reflecting a disease physiological state, specifically: Performing time domain, frequency domain and time-frequency domain analysis on the vital sign waveform data in the synchronized multi-modal data stream, extracting waveform morphology features, rhythm features and energy distribution features; Performing deep learning-based lesion region segmentation and quantification on the medical image sequence in the synchronized multi-modal data stream, extracting radiomics features, including texture features, shape features and intensity statistical features; Performing medical entity recognition and relationship extraction on the structured text records in the synchronized multi-modal data stream, constructing clinical event features, including diagnosis events, medication events and symptom events; The waveform morphology features, rhythm features, energy distribution features, radiomics features and clinical event features extracted for each modality are respectively summarized into feature sets for each corresponding modality.

[0008] Further, the feature sets extracted from different modalities are cross-modally fused to generate a patient-unified multi-dimensional time-series health status feature spectrum, specifically: Map the feature sets of different modalities to the same high-dimensional latent feature space; Calculate the correlation weights between different modality features in the high-dimensional latent feature space; According to the correlation weights, weight and fuse the features from different modalities but reflecting the same physiological or pathological dimension to form the fused dimension features; Arrange and combine all the fused dimension features according to the unified time axis index; The patient-unified multi-dimensional time-series health status feature spectrum is composed of all the fused dimension features at each time point.

[0009] Further, based on the multi-dimensional time-series health status feature spectrum, a clinical state evolution model representing the dynamic evolution of the patient's disease course is established in combination with a pre-set medical knowledge graph, specifically: Map the feature vector at each time point of the multi-dimensional time-series health status feature spectrum to a state node in the clinical state evolution model; According to the disease development path and state transition relationship defined in the medical knowledge graph, establish directed connection edges between state nodes at adjacent time points; Assign a weight to each directed connection edge, which is calculated based on the feature vector change amplitude and the typical mode of state transition in the medical knowledge graph; All state nodes and their weighted directed connection edges together constitute the clinical state evolution model describing the individualized disease course of the patient.

[0010] Further, the key clinical event nodes are defined in the clinical state evolution model, and the evolution distance of the current state of the patient from each key clinical event node is calculated, specifically: Predefined key clinical event nodes with important clinical significance are extracted from the medical knowledge graph, including disease deterioration turning points, complication occurrence points, and treatment response evaluation points; In the clinical state evolution model, the state nodes closest to the key clinical event nodes in the feature space are identified; The state node corresponding to the latest time point of the patient is taken as the current state node; The shortest path length from the current state node to the state node corresponding to each key clinical event node along the directed connection edge in the clinical state evolution model is calculated; The shortest path length is the evolution distance of the current state of the patient from each key clinical event node.

[0011] Further, the dynamic risk assessment surface of the patient evolving to different key clinical events is constructed according to the evolution distance and the preset evolution threshold, specifically: For each key clinical event node, set its corresponding evolution threshold; The evolution distance of the current state of the patient from each key clinical event node is divided by its corresponding evolution threshold to obtain the normalized relative evolution urgency; A multi-dimensional risk vector is constructed with different key clinical events as dimensions and relative evolution urgency as numerical values; The multi-dimensional risk vector is mapped to a continuous two-dimensional or three-dimensional space by a nonlinear function to form a dynamic risk assessment surface; Each point on the dynamic risk assessment surface represents a specific risk state combination, and the height or color represents the overall risk level.

[0012] Further, for the high-risk evolution path identified in the dynamic risk assessment surface, the multi-dimensional time-series health state feature spectrum is traced back to locate the key feature combination and retrieve the intervention measure evidence chain from the medical knowledge graph accordingly to generate a clinical decision support report, including: For the high-risk evolution path identified in the dynamic risk assessment surface, the multi-dimensional time-series health state feature spectrum is traced back to locate the key feature combination that leads to the risk increase; According to the key feature combination, the associated intervention measure evidence chain is retrieved from the medical knowledge graph; The structured clinical decision support report is generated by integrating the dynamic risk assessment surface, the high-risk evolution path, and the associated intervention measure evidence chain. According to the key feature combination, retrieve the associated intervention measure evidence chain from the medical knowledge graph, in particular: Input the key feature combination as a query condition into the medical knowledge graph; In the medical knowledge graph, match the pathophysiological state nodes that have direct or indirect causal or correlation relationship with the key feature combination; Retrieve the knowledge paths connecting the pathophysiological state nodes and various clinical intervention measure nodes; Sort and score the retrieved all knowledge paths by evidence level, and filter out several knowledge paths with the highest scores as recommended intervention measure evidence chains; The intervention measure evidence chain describes the logical reasoning chain from the identified key features to the specific intervention measures.

[0013] Further, the present application also includes a clinical auxiliary decision-making system based on integrated time-series multi-modal data, which comprises a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned clinical auxiliary decision-making method based on integrated time-series multi-modal data when executing the computer program.

[0014] Compared with the prior art, the present application has the following advantages: By calculating the evolution distance between the patient's current multi-dimensional time-series health status feature spectrum and the pre-defined key clinical event nodes, and constructing a dynamic risk assessment surface according to the evolution threshold. This technology integrates discrete time-series features into a continuous quantitative measure representing state evolution, mapping the patient's complex and dynamically changing clinical state into a computable evolution space. The dynamic risk assessment surface can simultaneously assess the instantaneous probability and changing trajectory of the patient's state towards multiple different clinical outcomes. This model overcomes the limitations of traditional static risk scoring models in the time dimension, and can provide continuous risk situation assessment as time passes. Clinicians can observe the continuous evolution trend of the patient's state based on the surface, and anticipate the possible development direction of the disease, so as to make a prediction within the key time window.

[0015] When a high-risk evolution path is identified, the system reverses back to trace the multi-dimensional time-series health status feature spectrum that generates the path, and locates the core feature combination that drives the risk evolution by analyzing the time-series correlation, change amplitude and pattern between the features. With the feature combination as the precise query condition, the system automatically retrieves the intervention measures, clinical guideline recommendations and research evidence that have strong medical logical association with it in the structured medical knowledge graph, forms a complete evidence chain, and generates a clinical report containing decision basis and recommendations accordingly. This process realizes the automatic link from risk identification to precise decision suggestion generation. It makes the high-risk warning output by the system directly associated with specific and explainable physiological feature abnormalities that lead to the risk, and establishes the decision suggestion on the basis of structured medical evidence. This link bridges the gap between risk warning and clinical action plan, reduces the time and cognitive load required for doctors to manually search and match evidence, and improves the efficiency and evidence-based reliability of the decision-making process. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A step graph of the clinical auxiliary decision-making method based on integrated time-series multi-modal data described in the present application; Figure 2 A flowchart for time alignment and missing value interpolation; Figure 3 A flowchart for multi-modal data feature analysis; Figure 4 A state node feature normalized Euclidean distance heat matrix; Figure 5 An evidence score and knowledge path hop number association graph for different acute kidney injury intervention measures. DETAILED DESCRIPTION

[0017] The technical solutions of the present application will be described below in conjunction with the embodiments, obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0018] Reference Figure 1, collect patient time-series multi-modal clinical data from different medical devices and information systems; time-align and impute missing values of the time-series multi-modal clinical data, and construct a synchronized multi-modal data stream with a unified time axis as a reference; perform feature analysis on each modality data in the synchronized multi-modal data stream respectively, and extract a feature set reflecting the physiological state of the disease; perform cross-modal fusion on the feature sets extracted from different modalities, and generate a patient unified multi-dimensional time-series health status feature spectrum; based on the multi-dimensional time-series health status feature spectrum, combine a preset medical knowledge graph, and establish a clinical state evolution model representing the dynamic evolution of the patient's disease course; define key clinical event nodes in the clinical state evolution model, and calculate the evolution distance between the patient's current state and each key clinical event node; according to the evolution distance and a preset evolution threshold, construct a dynamic risk assessment surface of the patient's evolution to different key clinical events; for the high-risk evolution path identified in the dynamic risk assessment surface, backtrack the multi-dimensional time-series health status feature spectrum, locate the key feature combination, and retrieve an intervention measure evidence chain from the medical knowledge graph accordingly, and generate a clinical decision support report.

[0019] In an embodiment of the present application, the collection of patient time-series multimodal clinical data from different medical devices and information systems is a process involving heterogeneous data source integration and annotation, the time-series multimodal clinical data at least including vital sign waveforms, medical image sequences and structured text records. Take a patient monitoring scenario of suspected septic shock in an intensive care unit (ICU) as an example, the collection of vital sign waveform data is achieved by connecting bedside monitoring devices through programmatic interfaces, the vital sign waveform data is continuously collected from the monitoring device interface, the electrocardiogram is collected at a frequency of 125 Hz and contains voltage signals of twelve leads, the blood oxygen saturation waveform reflects the light absorption changes of the pulsatile blood vessels, and the arterial blood pressure waveform is the pressure pulse curve measured by an arterial catheter. These waveform data are acquired in real time in a time resolution of milliseconds and are attached with the original time stamp of the device-generated data points. The collection of medical image sequences is completed through the image retrieval interface of the hospital's internal picture archiving and communication system, the medical image sequences of the patient are retrieved from the picture archiving and communication system in chronological order of examination time, for example, chest computed tomography sequences performed on the first day, the third day and the seventh day of the patient's admission, each examination of the computed tomography sequence contains dozens of axial slice images from the lung base to the lung top, and the magnetic resonance imaging sequence may include weighted imaging sequences for assessing tissue perfusion. Each sequence file is embedded with a standard medical digital imaging and communication protocol header file, which contains accurate examination date and time information. The collection of structured text records is achieved by querying the database interface of the hospital information system and the laboratory information system, the structured text records corresponding to the patient are extracted from the hospital information system and the laboratory information system, the order records provided by the hospital information system contain drug names, doses, administration routes and issuance times, the laboratory test reports provided by the laboratory information system contain numerical results such as white blood cell count, procalcitonin and lactic acid and sample collection time, and the nursing records contain regularly entered vital sign measurement values, consciousness scores and intake and output data. These records are stored in the relational database in the form of structured fields.

[0020] In some embodiments, annotating the original acquisition time point and data source modality type for each piece of time-series multi-modality clinical data is a necessary step to ensure data traceability. For vital sign waveform data continuously acquired from the monitoring device interface, each data packet is labeled with the acquisition time point synchronized from the device clock to the millisecond level, and the data source modality type is labeled as "continuous vital sign waveform". For medical image sequences retrieved from the picture archiving and communication system in chronological order of examination time, the "study date and time" field in the digital imaging and communications in medicine (DICOM) header file of each medical image sequence is extracted as the acquisition time point, and the data source modality type is labeled as "medical image sequence". For structured text records extracted from the hospital information system and the laboratory information system, the "record creation time" or "sample acquisition time" field associated with each record is extracted as the acquisition time point, and the data source modality type of the medical order record is labeled as "structured medical order text", the data source modality type of the laboratory test report is labeled as "structured test text", and the data source modality type of the nursing record is labeled as "structured nursing text". In this way, all the acquired original data have a unified time reference and modality identification.

[0021] It can be understood that there are inherent differences in the acquisition frequency and time granularity of different modalities of data. Vital sign waveform data continuously acquired from the monitoring device interface is a nearly continuous time series, medical image sequences retrieved from the picture archiving and communication system in chronological order of examination time are discrete time point events, and structured text records extracted from the hospital information system and the laboratory information system may be discrete or uneven in time distribution. This difference in time-series characteristics needs to be specially handled in subsequent steps. In a specific comparative example, the vital sign waveform data continuously acquired from the monitoring device interface of the same patient in the time period from 14:00 to 14:30 on October 26, 2023 provides about 7500 electrocardiogram sampling points per minute, blood oxygen saturation waveform and arterial blood pressure waveform data points, while the structured text records extracted from the hospital information system and the laboratory information system may only contain one laboratory test report with an acquisition time of 14:15 and several nursing record entries with intervals of several minutes, and the medical image sequences retrieved from the picture archiving and communication system in chronological order of examination time may have no data in this time period. This contrast in data density and periodicity highlights the asynchronicity and sparsity of multi-modality data in the time dimension.

[0022] Referring to Figure 2In one embodiment of the present invention, time alignment and missing value imputation of temporal multimodal clinical data are key steps in constructing a synchronized multimodal data stream. Taking the monitoring of a sepsis patient within 48 hours of admission as an example, the temporal multimodal clinical data includes vital sign waveform data continuously collected from the monitoring device interface, a chest computed tomography sequence retrieved from the image archiving and communication system, and structured text records extracted multiple times from the hospital information system and laboratory information system. The primary operation is to convert the timestamps of all data to the same standard time coordinate system. The device local time recorded in the electrocardiogram waveform data packet, the examination time in the medical digital imaging and communication protocol header file of the computed tomography sequence, and the acquisition time recorded in the laboratory test report database are all converted to the format "2023-10-27 10:00:00.000" based on Coordinated Universal Time (UTC), eliminating potential time zone differences or clock deviations between different medical devices and information systems.

[0023] For the converted time-series multimodal clinical data, a unified time axis index is established based on a preset time granularity. In the intensive care unit scenario, the time granularity can be set to one minute. Accordingly, a continuous time point sequence indexed in minutes is established for the 48 hours starting from the patient's admission time "2023-10-27 10:00:00", i.e., T1 (10:00), T2 (10:01)... up to T2880 (10:00 the next day). The integrity of each modality data at each time point in the time axis index is checked. For example, at the time point "2023-10-27 15:30:00", the system will check for the existence of vital sign waveform data, medical image sequence data, and structured text record data. Because computed tomography (CT) scan sequences are discrete events, medical image sequence data will be missing at most points in time; while vital sign waveform data should theoretically exist at every point in time, but may be missing due to momentary signal interruption; structured text records such as nursing entries may appear at specific points in time.

[0024] For time points with missing data, numerical interpolation is performed using interpolation based on adjacent time points with the same modality or extrapolation based on the correlation of cross-modal data. In practice, if the heart rate value is missing at a certain time point, but the heart rate values ​​for the previous and next minutes are available, linear interpolation is used to calculate the missing value. The interpolation process can be expressed by the following formula:

[0025] in: This represents the heart rate value that needs to be interpolated at time point t. Indicates the previous valid time point Heart rate value, represents the last valid time point heart rate value. For modalities such as medical image sequences that are completely missing in a long time interval, extrapolation based on cross-modal data correlation can be enabled, for example, when there is a lack of image data in a certain time period of the time axis index, but the vital sign waveform features and structured text records of inflammation indicators in that time period show high abnormality, the system can generate a quantitative feature estimation value indicating “suspected image abnormality aggravation” according to the cross-modal correlation model established in the historical training data, rather than directly generating pixel-level image data.

[0026] In some embodiments, the selection of time granularity can be adjusted according to clinical monitoring needs. For high-frequency physiological signal analysis, the time granularity can be set to seconds or even milliseconds, at which time the density of the time axis index is greatly increased, and the checking and interpolation algorithm for data integrity requires higher real-time performance. It can be understood that the preset time granularity determines the time resolution of the synchronized multi-modal data stream. A finer granularity can capture faster pathophysiological changes, but also correspondingly increases the computational complexity and the requirement for data interpolation reliability.

[0027] After completing the interpolation, each modality data is reorganized according to a unified time axis index to generate a synchronized multi-modal data stream corresponding to each time point. After reorganization, each time point index corresponds to a structured data packet. For example, the data packet corresponding to the time point “2023-10-27 22:05:00” contains the interpolated complete electrocardiogram waveform segment, blood oxygen saturation waveform segment, and arterial blood pressure waveform segment in that minute, marks the absence of original medical image sequence data at this time but possibly has extrapolated image feature estimation values, and records all structured text records such as a medical order of “give liquid resuscitation” and a nursing record of “central venous pressure measurement value is 8 mmHg” at this time point. Through this reorganization, the originally asynchronous and heterogeneous original multi-modal data in time is transformed into a neat data table or data stream that advances in uniform time steps.

[0028] Referring to Figure 3In an embodiment of the present application, feature parsing is performed on each modality data in the synchronized multi-modality data stream to extract feature sets reflecting the physiological state of the disease, and cross-modality fusion is performed on the feature sets extracted from different modalities to generate a unified multi-dimensional time-series health status feature spectrum of the patient, which constitutes the core conversion process from raw data to integrated feature representation. For the vital sign waveform data in the synchronized multi-modality data stream, time domain, frequency domain and time-frequency domain analysis are performed to extract waveform morphology features, rhythm features and energy distribution features. For example, for an electrocardiogram waveform segment, time domain analysis can calculate RR interval, QRS wave width, ST segment elevation and other waveform morphology features, frequency domain analysis can extract high frequency power and low frequency power in heart rate variability and other rhythm features through fast Fourier transform, and time-frequency domain analysis can depict the joint distribution features of the energy of the electrocardiogram signal in the time and frequency dimensions through wavelet transform. For the medical image sequence in the synchronized multi-modality data stream, lesion region segmentation and quantification based on deep learning are performed to extract image features, including texture features, shape features and intensity statistical features. For example, for a single time point image of a chest computed tomography sequence, a pre-trained U-Net network is used to segment the lung infection area, and then the contrast, correlation and other texture features of the gray level co-occurrence matrix are calculated from the segmented area, the volume, sphericity and other shape features of the lesion are calculated, and the mean, skewness and other intensity statistical features of the pixel values in the region are calculated. For the structured text records in the synchronized multi-modality data stream, medical entity recognition and relationship extraction are performed to construct clinical event features, including diagnosis events, medication events and symptom events, thereby constructing a structured "vasoactive drug use" medication event feature. The waveform morphology features, rhythm features, energy distribution features, image features and clinical event features extracted for each modality data are respectively summarized as feature sets of each corresponding modality, for example, all numerical features extracted from electrocardiogram, blood oxygen saturation waveform and arterial blood pressure waveform are combined into a vital sign modality feature set, all quantitative features extracted from computed tomography sequence and magnetic resonance imaging sequence are combined into a medical image modality feature set, and all event vectors parsed from medical order records, laboratory test reports and nursing records are combined into a text record modality feature set.

[0029] In some embodiments, cross-modal fusion of feature sets from different modalities begins by mapping these feature sets to the same high-dimensional latent feature space. This can be achieved by training a fully connected neural network encoder for each modal feature set, where each encoder transforms the original feature vector of its corresponding modality into a latent feature vector with the same dimension. Calculating the correlation weights between features from different modalities in the high-dimensional latent feature space is crucial for fusion. One implementation employs an attention-based weight calculation method. For feature vectors from different modalities that may reflect the same physiological dimension, their interaction attention scores are calculated to determine their contribution during fusion. The calculation of correlation weights can be expressed as the following formula:

[0030] in: Indicates the first A certain feature of the modality (as a query) and the first modality Normalized correlation weights between a certain feature (as a key) of a modality. It is the first Query vector projection of each modal feature It is the first Key vector projection of each modal feature Represents a vector similarity function (such as a dot product). It is the dimension of the key vector. This represents the total number of features involved in the calculation. Based on relevance weights, features from different modalities that reflect the same physiological or pathological dimension are weighted and fused to form fused dimensional features. For example, a dimension reflecting the risk of "circulatory failure" might be formed by weighted fusion of hypotension features from the vital signs modality, cardiac ejection fraction features from the medical imaging modality, and vasoactive drug use features from the text recording modality. All fused dimensional features are arranged and combined according to a unified timeline index. The patient's unified multi-dimensional temporal health status feature spectrum consists of all fused dimensional features at each time point, ultimately forming a three-dimensional data structure. Its three dimensions are the time point, the fused physiological / pathological dimension, and the quantified feature value of that dimension.

[0031] In an embodiment of the present application, a clinical state evolution model representing the dynamic evolution of the patient's disease course is established based on the multi-dimensional time-series health state feature spectrum combined with the preset medical knowledge graph, key clinical event nodes are defined in the clinical state evolution model, and the evolution distance of the patient's current state from each key clinical event node is calculated, and a dynamic risk assessment surface of the patient's evolution to different key clinical events is constructed according to the evolution distance and the preset evolution threshold. This series of steps realizes the transformation from static features to dynamic risk quantitative assessment. Specifically, the feature vector at each time point of the multi-dimensional time-series health state feature spectrum is mapped to a state node in the clinical state evolution model. For example, for a heart failure patient, the feature vector at time point T1 (at admission) may include a higher NT-proBNP value, a normal ejection fraction, and a slight dyspnea score. After normalization, this feature vector constitutes a state node named "S1" in the clinical state evolution model. The feature vector at time point T2 (6 hours after admission) may evolve into a further increased NT-proBNP value, a slightly decreased ejection fraction, and a moderate dyspnea score. This feature vector constitutes a state node named "S2". According to the disease development path and state transition relationship defined in the medical knowledge graph, a directed connection edge is established between the state nodes at adjacent time points. The medical knowledge graph may include state transition relationships such as "increased volume load" leading to "decreased cardiac output", and "decreased cardiac output" further leading to "end-organ hypoperfusion". If the change direction of the feature vector from S1 to S2 conforms to the typical feature change pattern of the state transition from "compensatory heart failure" to "decompensatory heart failure" in the medical knowledge graph, a directed connection edge from S1 to S2 is established between the state nodes S1 and S2. Each directed connection edge is assigned a weight, which is calculated based on the feature vector change amplitude and the typical pattern of state transition in the medical knowledge graph. The formula for calculating the weight can be designed as:

[0032] wherein: represents the weight of the directed connection edge from state node to state node , is the Euclidean distance between the feature vector of state node and the feature vector of state node , which quantifies the feature vector change amplitude, is a matching degree function, which measures the matching degree of the feature vector change with the pre-defined typical state transition pattern in the medical knowledge graph , the higher the matching degree, the larger the function value, is a harmonic coefficient between 0 and 1, used to balance the proportion of feature vector variation and prior knowledge of medical knowledge graph in weight calculation. All state nodes and weighted directed connection edges among them together constitute a clinical state evolution model describing the individualized disease course of the patient. The model is a weighted directed graph in structure, and the node sequence in the graph depicts the actual state evolution trajectory of the patient from admission to the current time.

[0033] Predefined key clinical event nodes with important clinical significance are extracted from the medical knowledge graph. The key clinical event nodes include disease deterioration turning points, complication occurrence points, and treatment response evaluation points. For example, in the medical knowledge graph related to heart failure, "acute pulmonary edema", "cardiogenic shock", "acute kidney injury", etc. can be predefined as disease deterioration turning points or complication occurrence points, and "good response to diuretic treatment" as a treatment response evaluation point. In the clinical state evolution model, the state node closest to the key clinical event node in the feature space is identified, i.e. the similarity between the feature vector of each state node in the clinical state evolution model and the standard feature vector of each key clinical event node is calculated, and the state node with the highest similarity is marked as the corresponding point of the key clinical event node in the model. The state node corresponding to the latest time point of the patient is taken as the current state node, and the shortest path length from the current state node to the state node corresponding to each key clinical event node along the directed connection edges in the clinical state evolution model is calculated. The shortest path length can be calculated using Dijkstra's algorithm, and the weights of all directed connection edges on the path are accumulated . The shortest path length is the evolution distance of the patient's current state to each key clinical event node. The numerical value of the evolution distance directly reflects the theoretical difficulty or "cost" required for the patient's current and historical state to evolve to a certain key clinical event. Refer to Table 1 for an exemplary calculation result.

[0034] Table 1: Key clinical event nodes and evolution distance table

[0035] For each key clinical event node, set its corresponding evolution threshold. The evolution threshold can be obtained from historical population data, for example, the evolution threshold of "acute pulmonary edema" event can be set to 20. Divide the evolution distance of the patient's current state to each key clinical event node by its corresponding evolution threshold to get the normalized relative evolution urgency. A relative evolution urgency less than 1 indicates that if the current evolution speed continues, the event node may be reached within the threshold time. Take different key clinical events as dimensions and the relative evolution urgency as numerical values to construct a multi-dimensional risk vector. For example, for the above four key clinical events, a four-dimensional risk vector can be constructed The multi-dimensional risk vector is mapped to a continuous two-dimensional or three-dimensional space by a non-linear function to form a dynamic risk assessment surface. One mapping method is to reduce the four-dimensional risk vector to two dimensions by principal component analysis, and then generate a continuous surface by kernel density estimation. Each point on the dynamic risk assessment surface represents a risk state combination, and the height or color represents the overall risk level.

[0036] Referring to Figure 4 , the pairwise similarity measures of the six patient health state nodes S0 to S5 in the high-dimensional feature space are presented, and the values and corresponding heat chromatography of the upper and lower triangular symmetric distribution matrices accurately depict the normalized Euclidean distance between the starting state node and the target state node: the closer the value is to 0 (the chromatography is bright yellow), the smaller the difference between the multi-dimensional time series health feature spectrum of the two nodes, and the closer the pathophysiological state, such as the distance between S4 and S5 is 0.32, and the distance between S2 and S3 is 0.38; the closer the value is to 2.0 (the chromatography is dark burgundy), the more significant the state difference, and the greater the pathophysiological state evolution span, such as the distance between S0 and S5 is 2.0, and the distance between S0 and S4 is 1.68. In the construction process of the clinical state evolution model, the values of the matrix directly provide the core parameters, the feature vector change amplitude, for the calculation of the directed connection edge weight. Specifically, each non-diagonal element in the matrix can be used as a quantitative value of the weight calculation formula , combined with the pre-defined state transition matching degree function and the harmonic coefficient , the weight assignment of the transition edge between each state node can be completed. The normalized distance matrix also provides a similarity judgment benchmark for the matching of key clinical event nodes, which can assist in identifying the model state node closest to the pre-set key clinical event (such as the turning point of disease deterioration), and laying a standardized feature difference measurement foundation for subsequent evolution distance calculation and dynamic risk assessment surface construction.

[0037] In an embodiment of the present application, for the high-risk evolution path identified in the dynamic risk assessment surface, the multi-dimensional time-series health status feature spectrum is backtracked to locate the key feature combination, and the clinical decision support report is finally generated from the intervention measure evidence chain retrieved from the medical knowledge graph according to the key feature combination, which completes the closed loop from risk early warning to clinical decision suggestion. The high-risk evolution path identified in the dynamic risk assessment surface may be a path leading to the key clinical event node of "acute kidney injury", and the relative evolution urgency of the path jumps from 0.5 to 0.9 in a short time. The system immediately starts the backtracking analysis program, backtracks the multi-dimensional time-series health status feature spectrum to locate the key feature combination leading to the risk increase, and backtracks the analysis along the time axis to review the fusion dimension feature values corresponding to the recent state nodes constituting the high-risk path. By comparing the differences between the feature spectrum before and after the risk jump, the key feature combination is located as the "urine volume" feature value in the "renal function" dimension continuously decreasing, the "serum creatinine" feature value sharply rising, the "mean arterial pressure" feature value in the "hemodynamics" dimension slowly decreasing, and the "white blood cell count" feature value in the "inflammation" dimension maintaining a high level. The synergistic abnormal changes of this group of feature values are identified as the key feature combination.

[0038] Based on key feature combinations, evidence chains of related interventions are retrieved from the medical knowledge graph. These key feature combinations are input as query conditions into the medical knowledge graph, where they are parsed into a series of entities and relational assertions, such as "decreased urine output," "elevated serum creatinine," "decreased mean arterial pressure," and "elevated white blood cell count." Pathophysiological state nodes with direct or indirect causal or correlational relationships to the key feature combinations are matched within the medical knowledge graph. For example, "decreased urine output" and "elevated serum creatinine" might both point to the "acute kidney injury" state node, "decreased mean arterial pressure" might be associated with the "inadequate renal perfusion" state node, and "elevated white blood cell count" might be associated with the "systemic inflammatory response" state node. Knowledge paths connecting pathophysiological state nodes and various clinical intervention nodes are retrieved. Starting from the "acute kidney injury" and "inadequate renal perfusion" state nodes, the medical knowledge graph might contain intervention nodes such as "optimize volume status," "avoid nephrotoxic drugs," and "consider renal replacement therapy," along with relational edges connecting these states and interventions such as "recommended for" and "may be beneficial." All retrieved knowledge paths are ranked by evidence level and scored by relevance. The top-scoring knowledge paths are selected as the recommended intervention evidence chains. The evidence level ranking is based on the source of medical evidence involved in the path, such as randomized controlled trials, cohort studies, or expert consensus, and different weights are assigned accordingly. The relevance score comprehensively considers the semantic similarity between query features and nodes in the path, as well as the path length. The intervention evidence chain describes the logical reasoning chain from the identified key features to the specific intervention. A structured clinical decision support report is generated by integrating the dynamic risk assessment surface, high-risk evolution paths, and associated intervention evidence chains. The report is presented in a standardized format and includes patient identification, risk warning summary, key feature combinations and their time-series changes identified through retrospective analysis, the recommended intervention evidence chain, and an overview of its supporting evidence.

[0039] In some embodiments, the specific calculation of evidence level ranking and relevance score for all retrieved knowledge paths can employ a combination of graph-based ranking algorithms and rule-weighted methods. The formula for calculating the relevance score can be defined as follows:

[0040] in: Represents a knowledge path The final score, It is a path The set of all relation edges traversed. It is based on the relation edge The pre-set weights for the associated evidence levels (e.g., Level I, Level II, Level III evidence), It is an indicator function that takes the value 0 or 1, when the relation edge... 1 if the type of the query feature is highly relevant to the context of the combination of the query feature and the key feature, otherwise 0, is the number of hops of the path is the number of hops of the path (i.e. the number of nodes minus one), is a tuning parameter for adjusting the impact of the path length. The formula encourages the selection of knowledge paths that pass through strong correlations of high evidence level and are concise.

[0041] It can be understood that the positioning of the key feature combination is not limited to the feature value at a single time point, but more concerned about the change trend and pattern of the feature in the risk jump time window. The completeness and quality of the medical knowledge graph directly determine the clinical rationality and practicality of the retrieved intervention measure evidence chain. The structured clinical decision support report does not replace the judgment of the clinician, but provides integrated information traced and reasoned to assist decision-making. Optionally, when generating the clinical decision support report, a visual screenshot of the dynamic risk assessment surface and a position diagram of the high-risk evolution path in the clinical state evolution model can be attached to enhance the interpretability of the report. When retrieving the intervention measure evidence chain, in addition to the pre-defined medical knowledge graph, online clinical guideline databases or the latest medical literature abstracts can also be accessed to supplement the evidence chain in real time by natural language processing technology. For several screened intervention measure evidence chains, secondary sorting and merging can be performed according to their potential synergies or mutual contraindications to form a coordinated and prioritized intervention measure suggestion list.

[0042] Referring to Figure 5, the evidence scores of different clinical interventions and the corresponding knowledge path hops are shown, specifically, the evidence scores (0-1) of 'optimizing volume status', 'avoiding nephrotoxic drugs','renal replacement therapy', 'anti-inflammatory therapy' and 'vasopressor therapy' are shown in the form of column chart, and the score values are 0.95, 0.92, 0.88, 0.85 and 0.80 respectively, and the red line represents the hops of the corresponding knowledge path in the medical knowledge graph, and the range is 2.0-3.0. From the figure, it can be seen that the evidence scores of 'optimizing volume status' and 'avoiding nephrotoxic drugs' are the highest, which are 0.95 and 0.92 respectively, and the corresponding knowledge path hops are the lowest (about 2.0), which indicates that the evidence sources are high-level I-class RCT evidence, and the knowledge path is simple, the reasoning chain is short, which meets the optimal score principle of 'high evidence level + strong correlation + short path' in the formula. The knowledge path hops of'renal replacement therapy' and 'vasopressor therapy' are increased to 3.0, although the evidence score of the former is still at a high level (0.88, II-class cohort study), but the increase of the path hops reflects that the reasoning chain is longer and more intermediate nodes are involved. The evidence score (0.85, II-class cohort study) and path hop (about 2.0) of 'anti-inflammatory therapy' are at a medium level, and the evidence score of 'vasopressor therapy' is the lowest (0.80, III-class expert consensus), but the path hop is also 3.0, which reflects the combination characteristics of low evidence level and long path.

[0043] The above examples are only used to illustrate the technical method of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical method of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical method of the present application.

Claims

1. A clinical decision support method based on integrated time-series multimodal data, characterized in that, Includes the following steps: Collect time-series multimodal clinical data from patients from different medical devices and information systems; The time-series multimodal clinical data are time-aligned and missing values ​​are imputed to construct a synchronized multimodal data stream based on a unified time axis; For each modality of data in the synchronized multimodal data stream, feature parsing is performed to extract a feature set reflecting the physiological state of the disease; The feature sets extracted from different modalities are fused across modalities to generate a unified multi-dimensional temporal health status feature spectrum for patients. Based on the multi-dimensional temporal health status feature spectrum, combined with a pre-set medical knowledge graph, a clinical state evolution model representing the dynamic evolution of the patient's disease course is established. Key clinical event nodes are defined in the clinical state evolution model, and the evolution distance between the patient's current state and each key clinical event node is calculated. Based on the evolution distance and the preset evolution threshold, a dynamic risk assessment surface is constructed to show the patient's evolution toward different key clinical events; For the high-risk evolution path identified in the dynamic risk assessment surface, the multi-dimensional time-series health status feature spectrum is traced back to locate key feature combinations and, based on this, the evidence chain of intervention measures is retrieved from the medical knowledge graph to generate a clinical decision support report.

2. The clinical decision support method based on integrated time-series multimodal data according to claim 1, characterized in that, The collection of patient time-series multimodal clinical data from different medical devices and information systems specifically includes: The time-series multimodal clinical data includes at least vital sign waveforms, medical image sequences, and structured text records; Vital signs waveform data are continuously collected from the monitoring device interface, including electrocardiogram, blood oxygen saturation waveform, and arterial blood pressure waveform; The patient's medical image sequence is retrieved from the image archiving and communication system in chronological order of examination time. The medical image sequence includes computed tomography (CT) scan sequence and magnetic resonance imaging (MRI) sequence. Structured text records corresponding to patients are extracted from the hospital information system and laboratory information system. These structured text records include medical orders, laboratory test reports, and nursing records. Each piece of the aforementioned time-series multimodal clinical data is labeled with its original acquisition time point and data source modality type.

3. A clinical decision support method based on integrated time-series multimodal data as described in claim 2, characterized in that, The process of performing time alignment and missing value imputation on the temporal multimodal clinical data to construct a synchronized multimodal data stream based on a unified time axis specifically involves: The timestamps of all data in the aforementioned time-series multimodal clinical data are converted to the same standard time coordinate system; For the converted time-series multimodal clinical data, a unified time axis index is established based on a preset time granularity; Check the integrity of each modality data at each time point in the time axis index; For time points where data is missing, numerical interpolation is performed using interpolation based on the same modal data of adjacent time points or extrapolation based on the correlation of cross-modal data. After interpolation, the modal data are reorganized according to the unified time axis index to generate a synchronized multimodal data stream with one time point corresponding to each other.

4. A clinical decision support method based on integrated temporal multimodal data according to claim 3, characterized in that, The step of performing feature parsing on each modality of data in the synchronized multimodal data stream to extract a feature set reflecting the physiological state of the disease specifically involves: Time-domain, frequency-domain, and time-frequency-domain analyses are performed on the vital sign waveform data in the synchronized multimodal data stream to extract waveform morphology features, rhythm features, and energy distribution features. The medical image sequence in the synchronized multimodal data stream is segmented and quantized based on deep learning, and radiomics features, including texture features, shape features and intensity statistical features, are extracted. Medical entity recognition and relation extraction are performed on the structured text records in the synchronized multimodal data stream to construct clinical event features, including diagnostic events, medication events, and symptom events. The waveform morphology features, rhythm features, energy distribution features, radiomics features, and clinical event features extracted for each modality of data are summarized into feature sets for each corresponding modality.

5. A clinical decision support method based on integrated temporal multimodal data according to claim 4, characterized in that, The step of fusing the feature sets extracted from different modalities across modalities to generate a unified multi-dimensional temporal health status feature spectrum for the patient is specifically as follows: The feature sets of different modalities are mapped to the same high-dimensional latent feature space; In the high-dimensional latent feature space, the correlation weights between different modal features are calculated; Based on the aforementioned correlation weights, features from different modalities but reflecting the same physiological or pathological dimension are weighted and fused to form fused dimensional features. All the fused dimensional features are arranged and combined according to the unified time axis index; The unified multidimensional temporal health status feature spectrum of the patient is composed of all fused dimensional features at each time point.

6. A clinical decision support method based on integrated temporal multimodal data according to claim 5, characterized in that, Based on the multi-dimensional temporal health status feature spectrum and combined with a pre-set medical knowledge graph, a clinical state evolution model representing the dynamic evolution of the patient's disease course is established, specifically as follows: The feature vector of each time point in the multi-dimensional temporal health status feature spectrum is mapped to a state node in the clinical state evolution model. Based on the disease development path and state transition relationship defined in the medical knowledge graph, directed connection edges are established between state nodes at adjacent time points; Each directed connection edge is assigned a weight, which is calculated based on the magnitude of feature vector change and typical state transition patterns in the medical knowledge graph; All state nodes and the weighted directed edges between them together constitute a clinical state evolution model describing the individualized course of a patient's disease.

7. A clinical decision support method based on integrated temporal multimodal data according to claim 6, characterized in that, The process involves defining key clinical event nodes in the clinical state evolution model and calculating the evolution distance between the patient's current state and each key clinical event node, specifically as follows: Predefined key clinical event nodes with significant clinical importance are extracted from the medical knowledge graph. These key clinical event nodes include turning points in disease deterioration, points where complications occur, and points for evaluating treatment response. In the clinical state evolution model, the state node that is closest to the key clinical event node in the feature space is identified; The current state node is the state node corresponding to the patient's latest time point. Calculate the shortest path length from the current state node, along the directed connection edges in the clinical state evolution model, to the state node corresponding to each key clinical event node; The shortest path length is the distance between the patient's current state and each key clinical event node.

8. A clinical decision support method based on integrated temporal multimodal data according to claim 7, characterized in that, The process involves constructing a dynamic risk assessment surface for the patient's progression towards different key clinical events based on the evolution distance and a preset evolution threshold, specifically as follows: For each key clinical event node, a corresponding evolution threshold is set; Divide the distance between the patient’s current state and each key clinical event node by its corresponding evolution threshold to obtain the normalized relative evolution urgency. A multidimensional risk vector is constructed using different key clinical events as dimensions and relative urgency as a numerical value. The multidimensional risk vector is mapped to a continuous two-dimensional or three-dimensional space by a nonlinear function, forming a dynamic risk assessment surface. Each point on the dynamic risk assessment surface represents a specific combination of risk states, and its height or color indicates the overall risk level.

9. A clinical decision support method based on integrated time-series multimodal data according to claim 8, characterized in that, The process involves identifying high-risk evolution paths in the dynamic risk assessment surface, tracing back the multi-dimensional time-series health status feature spectrum, locating key feature combinations, and retrieving the evidence chain of intervention measures from the medical knowledge graph based on these combinations to generate a clinical decision support report, including: For the high-risk evolution path identified in the dynamic risk assessment surface, the multi-dimensional time-series health status feature spectrum is traced back to locate the key feature combination that leads to increased risk. Based on the combination of key features, retrieve the associated chain of evidence for intervention measures from the medical knowledge graph; By integrating the dynamic risk assessment surface, high-risk evolution path, and the evidence chain of associated intervention measures, a structured clinical decision support report is generated. The step of retrieving the associated evidence chain of intervention measures from the medical knowledge graph based on the combination of key features specifically involves: The key features are combined and used as query conditions, which are then input into the medical knowledge graph. In the medical knowledge graph, match the pathophysiological state nodes that have a direct or indirect causal or correlational relationship with the combination of the key features; Retrieve the knowledge paths that connect the pathophysiological state nodes with various clinical intervention nodes; All retrieved knowledge paths were ranked by evidence level and scored by relevance. The top-scoring knowledge paths were selected as the evidence chain for recommended intervention measures. The chain of evidence for intervention measures describes the logical reasoning chain from the identified key features to the specific intervention measures.

10. A clinical decision support system based on integrated temporal multimodal data, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a clinical decision support method based on integrated temporal multimodal data as described in any one of claims 1 to 9.

Images