A cerebral hemorrhage prognosis prediction method and system based on multi-modal data fusion

Abstract

The application relates to the technical field of medical information processing, in particular to a cerebral hemorrhage prognosis prediction method and system based on multi-modal data fusion, which comprises a data acquisition and preprocessing module, a feature extraction module and a prediction reasoning module; the data acquisition and preprocessing module is used for constructing a multi-modal time sequence correlation data group; the feature extraction module is used for extracting an image feature set, a clinical feature set and a time sequence dynamic feature set from the multi-modal time sequence correlation data group; the prediction reasoning module is used for generating a prognosis comprehensive report, and judging whether a patient follows the prognosis comprehensive report to perform various prognosis treatments according to the latest multi-modal data of the patient and the prognosis comprehensive report. Through the design of the feature extraction module and the prediction reasoning module, various multi-modal data of the patient in the whole course can be analyzed, the condition change of the patient can be comprehensively judged, the prognosis result of the patient can be more reasonably predicted, individualized prognosis treatment suggestions including operation timing, complication prevention and rehabilitation management are generated, and closed-loop feedback is realized.

Description

Technical Field

[0001] This invention relates to the field of medical information processing technology, and in particular to a prognostic prediction and clinical decision support technology for cerebral hemorrhage based on artificial intelligence and medical imaging, specifically to a prognostic prediction method and system for cerebral hemorrhage based on multimodal data fusion. Background Technology

[0002] Intracerebral hemorrhage (ICH) refers to the non-traumatic rupture of blood vessels within the brain, leading to the accumulation of blood in the brain parenchyma. It is a severe subtype of stroke. Statistics show that ICH accounts for approximately 10% to 30% of all strokes, but its disability and mortality rates are far higher than those of ischemic stroke. ICH has a rapid onset and progression; patients often experience progressive worsening of neurological deficits within minutes to hours of onset. Furthermore, the frequent occurrence of complications such as hematoma expansion, cerebral edema, and hydrocephalus makes the clinical management of ICH extremely time-sensitive and requires precise decision-making.

[0003] Existing prognostic prediction systems for cerebral hemorrhage utilize deep learning algorithms to automatically analyze medical images, extracting high-dimensional features such as hematoma morphology and density distribution. These features are then integrated with structured information including clinical laboratory data and vital signs to perform risk stratification analysis, thereby predicting the prognosis of patients with cerebral hemorrhage. The main approaches include: 1. An ICH Score / ICHOP scoring system based on CT images, combined with indicators such as GCS, hematoma volume, and intraventricular hemorrhage to predict mortality; 2. An automatic hematoma segmentation and hematoma expansion prediction method based on deep learning (3D-CNN); 3. A machine learning prognostic prediction model based on electronic health data (EHR).

[0004] However, in practical applications, numerous factors influence the prognosis of patients with cerebral hemorrhage, such as their physical condition before admission, the treatment plan during hospitalization, and their recovery after discharge. Current technologies rely solely on medical imaging during hospitalization to analyze patient prognosis, making it difficult to gain a comprehensive understanding of the patient's condition and potentially leading to inaccurate predictions. Furthermore, existing scoring systems depend on static admission indicators, making it difficult to capture the dynamic evolution of the disease course; single imaging or clinical models provide insufficient information to integrate treatment intervention information; closed-loop management from prediction to treatment recommendations, implementation supervision, and feedback verification is lacking; and early prediction of surgical / conservative treatment branches and postoperative complications (cerebral edema and hydrocephalus) is also lacking. Therefore, it is necessary to propose a prognostic prediction method and system for cerebral hemorrhage based on multimodal data fusion to address these issues. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method and system for predicting the prognosis of cerebral hemorrhage based on multimodal data fusion. Through the design of data acquisition and preprocessing modules, various data of the patient at different time periods can be collected based on the patient's admission time. Through the design of feature extraction and prediction inference modules, the various data are analyzed to comprehensively judge the changes in the patient's condition, thereby more reasonably predicting the patient's prognosis.

[0006] This invention has the following key technical points:

[0007] (1) Multimodal temporal alignment and disease stage aggregation; (2) 3D-CNN image feature extraction and CT sign recognition; (3) LSTM biochemical index temporal encoding; (4) Cross-disease-stage temporal dynamic features (change slope / extreme difference / fluctuation entropy); (5) Multimodal feature adaptive weighted fusion; (6) Clinical decision rule base individualized suggestion generation; (7) Task scheduling and execution supervision; (8) 90-day follow-up online incremental learning feedback.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows: a prognostic prediction system for cerebral hemorrhage based on multimodal data fusion, comprising:

[0009] The data acquisition and preprocessing module is used to acquire patients' multimodal data and align the multimodal data to a unified time axis according to the acquisition time to construct a multimodal time-series correlated data group; the multimodal data includes at least imaging data, clinical laboratory data, medical history and physical examination data, and treatment intervention data.

[0010] The key time points in the timeline include at least the time of admission (t=0), within 6 hours of admission, within 24 hours of onset, within 72 hours of admission, the time of discharge, and 90 days after discharge.

[0011] The feature extraction module is used to receive multimodal time-series correlated data sets and extract radiological feature sets, clinical feature sets, and time-series dynamic feature sets from the multimodal time-series correlated data sets.

[0012] The prediction and inference module receives imaging feature sets, clinical feature sets, and temporal dynamic feature sets. It also constructs a multimodal fusion prediction model using a convolutional neural network model. The multimodal fusion prediction model uses a cross-modal attention mechanism to perform weighted fusion of the imaging feature sets, clinical feature sets, and temporal dynamic feature sets to obtain a fused feature vector. Based on the fused feature vector, it uses SHAP values ​​to perform feature importance analysis and generate prognostic prediction results.

[0013] The prediction output module receives the prognostic prediction results, analyzes the corresponding prognostic treatment suggestions based on the prognostic prediction results, generates a comprehensive prognostic report based on the prognostic prediction results and prognostic treatment suggestions, and determines whether the patient has followed the comprehensive prognostic report to perform various prognostic treatments based on the patient's latest multimodal data and the comprehensive prognostic report. Based on the execution status of each prognostic treatment, a task scheduling report is generated.

[0014] The data management module is used to store, query, and update comprehensive prognostic reports for patients throughout their entire disease course.

[0015] Furthermore, the specific method for aligning the execution time of the data acquisition and preprocessing modules is as follows:

[0016] Based on the admission time, the acquisition time of multimodal data is converted into an offset relative to the admission time; each multimodal data is divided into corresponding disease course stage groups according to a preset time window, including the admission stage, hospitalization stage, and discharge stage; multiple data points of the same modality within the same disease course stage are time-series aggregated to generate representative feature values ​​of that modality in that stage.

[0017] Furthermore, the prognostic prediction results should include at least the risk level of poor prognosis, the risk level of disease progression, and the risk level of complications; the risk level of disease progression includes the risk level of hematoma expansion, the risk level of cerebral edema, and the risk level of hydrocephalus.

[0018] Furthermore, the feature extraction module includes:

[0019] The image feature extraction unit is used to segment the hematoma region through a three-dimensional convolutional neural network, calculate the hematoma volume, morphological features and density distribution features, and perform differential calculation on image features at different time series to generate dynamic evolution features of the hematoma; the prediction and reasoning module is used to determine the risk level of poor prognosis based on the dynamic evolution features of the hematoma.

[0020] Furthermore, the feature extraction module also includes:

[0021] The clinical feature extraction unit is used to encode the temporal change trend of biochemical test indicators using a long short-term memory network, extract baseline feature vectors and treatment strategy feature vectors, and then fuse and encode them through a multilayer perceptron to generate a clinical feature set; the prediction and reasoning module is used to determine the risk level of complications based on the clinical feature set.

[0022] Furthermore, the feature extraction module also includes:

[0023] The temporal feature extraction unit is used to calculate the slope, extreme value difference, and fluctuation entropy of the numerical changes of the same feature type across disease stages, and to obtain data on changes in hematoma volume, edema volume, midline shift, ventricular volume, and Evans index, generating a temporal dynamic feature set. The prediction and inference module is used to determine the risk level of hematoma expansion based on changes in hematoma volume, the risk level of cerebral edema based on changes in edema volume and midline shift, and the risk level of hydrocephalus based on changes in ventricular volume and Evans index. The risk levels of hematoma expansion, cerebral edema, and hydrocephalus are all categorized as high, medium, and low risk.

[0024] Furthermore, the prediction output module includes:

[0025] The decision suggestion generation unit is used to input historical prognostic results and their treatment plans, construct a clinical decision rule base based on historical prognostic results and their treatment plans, and generate prognostic suggestions based on prognostic prediction results and the clinical decision rule base. The prognostic suggestions include at least the timing of surgery, complication prevention measures, and rehabilitation management plans.

[0026] Furthermore, the prediction output module also includes:

[0027] The task scheduling unit generates an executed list based on the executed pre-processing content and generates an unexecuted reminder report based on the unexecuted pre-processing content.

[0028] Furthermore, it also includes:

[0029] The feedback verification module is used to collect multimodal data of patients 90 days after actual discharge as the actual prognostic result. It compares the predicted prognostic result with the actual prognostic result to obtain the comparison result, and feeds the comparison result back to the prediction inference module. The prediction inference module is used to update the multimodal fusion prediction model based on the comparison result.

[0030] The technical principle of the above solution is as follows:

[0031] Through the data acquisition and preprocessing module, using the patient's admission time as the time benchmark, imaging data, clinical laboratory data, medical history and physical examination data, and treatment intervention data collected at different time points are uniformly mapped to a standard time axis and divided into admission, hospitalization and discharge stages according to preset time windows. By performing time-series aggregation on multiple data points of the same modality within the same disease stage, data deviations caused by asynchronous acquisition are eliminated, forming a structured multimodal time-series associated data set.

[0032] The system will trigger differentiated prediction tasks at different time windows. Within 6 hours of patient admission, the system will predict hematoma expansion, make surgical decisions 72 hours after patient admission, continuously monitor cerebral edema and hydrocephalus during hospitalization, and conduct adverse prognosis assessment 90 days after discharge.

[0033] The feature extraction module uses a 3D convolutional neural network to segment hematoma and extract dynamic evolution features from image data. It also encodes the temporal trends of clinical laboratory indicators using a long short-term memory network and calculates the slope of numerical changes, extreme value differences, and fluctuation entropy across disease stages as temporal dynamic features. This achieves a multi-dimensional quantitative expression of image spatial features, clinical temporal features, and disease evolution features. The prediction and inference module constructs a multimodal fusion prediction model based on a convolutional neural network. It performs cross-modal weighted fusion of these three feature sets to obtain a fused feature vector, fully utilizing the complementarity between different modal information to output prognostic prediction results including the risk level of poor prognosis, the risk level of disease progression (hematoma expansion risk level, cerebral edema risk level, hydrocephalus risk level), and the risk level of complications.

[0034] The prediction output module further integrates a clinical decision rule base built from historical prognostic results and their treatment plans, transforming the prediction results into specific prognostic treatment suggestions such as surgical timing recommendations, complication prevention measures, and rehabilitation management plans, and generating a comprehensive prognostic report. At the same time, it tracks the implementation status of various prognostic treatments based on the patient's latest multimodal data, generating an executed list and an unexecuted reminder report, thus achieving closed-loop management from prediction to execution.

[0035] The feedback verification module collects prognostic outcome data from patients 90 days after actual discharge, compares the predicted results with the actual results, and sends the comparison results back to the prediction inference module. This drives the online incremental learning and updating of the multimodal fusion prediction model, enabling the system's predictive ability to continuously optimize with data accumulation. The above approach offers the following benefits:

[0036] 1. This invention, through the design of its data acquisition and preprocessing modules, uses the patient's admission time as a benchmark to uniformly align scattered imaging data, clinical laboratory data, medical history and physical examination data, and treatment intervention data onto a standard timeline. It then groups and aggregates these data according to admission, hospitalization, and discharge stages, thereby gaining a comprehensive understanding of all aspects of the patient's disease progression. This provides a detailed and complete data foundation for subsequent multimodal fusion prediction, ensuring the feasibility and comprehensiveness of predictive analysis. This invention significantly improves the timeliness of early intervention by performing differentiated predictions within key time windows (6h / 24h / 72h / 90 days).

[0037] 2. This invention, through the design of a feature extraction module, can extract features from various types of data, improving the efficiency of subsequent data analysis. Then, through a prediction and inference module, it performs weighted fusion based on the feature sets of various data to obtain a fused feature vector, thereby comprehensively judging the patient's condition. Compared with the existing technology that relies on only a single data for analysis, this invention utilizes the complementary and synergistic relationship between different modal information to judge the risk level of poor prognosis, the risk level of disease progression, and the risk level of complications, overcoming the problems of insufficient information from a single data source and the strong subjectivity of prediction results, and significantly improving the accuracy and reliability of prognosis prediction.

[0038] 3. This invention, through the decision suggestion generation unit in the prediction output module, combines the prognostic prediction results with a clinical decision rule base constructed from historical prognostic results and their treatment plans, automatically transforming them into individualized prognostic treatment suggestions such as surgical timing recommendations, complication prevention measures, and rehabilitation management plans, and generating a structured comprehensive prognostic report. Compared with existing technologies that only provide prediction results, this invention can provide doctors with effective decision-making assistance and reduce the difficulty of clinical decision-making. At the same time, through the task scheduling unit, it tracks the execution status of various prognostic treatments, generates an executed list and an unexecuted reminder report, realizing full-process supervision from prediction to execution, and effectively improving patients' compliance with clinical intervention plans.

[0039] 4. This invention reduces the difficulty of clinical decision-making by providing decision support for surgical and conservative treatment options, and improves clinical usability through a structured reporting system (image report / biochemical report / risk report / follow-up report).

[0040] 5. This invention utilizes an online incremental learning feedback mechanism to continuously optimize the system's predictive ability as data accumulates. Furthermore, a method for predicting the prognosis of cerebral hemorrhage based on multimodal data fusion includes the following steps:

[0041] S1, Data Acquisition: Acquire the patient's current multimodal data, align the multimodal data to a unified time axis according to the acquisition time, and construct a multimodal time-series correlated data group.

[0042] S2, Feature Extraction: Extracting radiological feature sets, clinical feature sets, and time-series dynamic feature sets from multimodal temporal correlation datasets.

[0043] S3, Multimodal Fusion Prediction: Input the imaging feature set, clinical feature set, and temporal dynamic feature set into the multimodal fusion prediction model.

[0044] S4, Prognostic Management Recommendation Generation: Obtain the comprehensive prognostic report and output the prognostic prediction results and prognostic management recommendations from the comprehensive prognostic report.

[0045] S5, Task Scheduling and Execution Supervision: Collects the patient's current prognostic treatment content and execution status, generates an executed list based on the executed prognostic treatment content, and generates an unexecuted reminder report based on the unexecuted prognostic treatment content.

[0046] S6, Feedback Validation and Model Update: Collect multimodal data of patients 90 days after discharge and feed the multimodal data of patients 90 days after discharge back to the multimodal fusion prediction model to update the multimodal fusion prediction model.

[0047] Beneficial effects: This approach can effectively analyze data from patients with cerebral hemorrhage at different times and of different types, thereby improving the effectiveness and rationality of prognostic outcome analysis and prognostic management recommendations, and thus optimizing the patient's prognosis. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the prognostic prediction system for cerebral hemorrhage based on multimodal data fusion according to the present invention.

[0049] Figure 2 This is a schematic diagram of the feature extraction module in the prognostic prediction system for cerebral hemorrhage based on multimodal data fusion of the present invention.

[0050] Figure 3 This is a schematic diagram of the prediction output module in the prognostic prediction system for cerebral hemorrhage based on multimodal data fusion of the present invention.

[0051] Figure 4 This is a schematic diagram illustrating the steps of the prognostic prediction method for cerebral hemorrhage based on multimodal data fusion according to the present invention. Detailed Implementation

[0052] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the method or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0054] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0055] The following detailed description illustrates the specific implementation methods:

[0056] Example 1:

[0057] like Figure 1 As shown, a prognostic prediction system for cerebral hemorrhage based on multimodal data fusion includes a data acquisition and preprocessing module, a feature extraction module, a prediction inference module, a prediction output module, a data management module, and a feedback verification module.

[0058] The specific functions of each module are as follows:

[0059] The data acquisition and preprocessing module is used to acquire patients' multimodal data (in this embodiment, the specific scope of multimodal data is as follows: demographic data, past medical history, disease information, admission clinical data (GCS / NIHSS / ICH Score), imaging data (CT / CTA / MR), biochemical data (complete blood count / 21 biochemical items / coagulation and fibrinolysis), surgical treatment data, and non-surgical treatment data), and align the multimodal data to a unified time axis according to the acquisition time to construct a multimodal time-series associated data group; the multimodal data includes at least imaging data, clinical laboratory data, medical history and physical examination data, and treatment intervention data.

[0060] The specific method for aligning the execution time of the data acquisition and preprocessing modules is as follows:

[0061] Based on the admission time, the acquisition time of multimodal data is converted into an offset relative to the admission time; each multimodal data is divided into corresponding disease course stage groups according to a preset time window, including the admission stage, hospitalization stage, and discharge stage; multiple data points of the same modality within the same disease course stage are time-series aggregated to generate representative feature values ​​of that modality in that stage.

[0062] Specifically, in this embodiment, key time points in the timeline include admission (t=0), within 6 hours of admission, within 24 hours of onset, within 72 hours of admission, discharge, and 90 days after discharge. The system will trigger different prediction tasks at different time windows: predicting the risk of hematoma expansion at t<=6h, assessing surgical recommendations at t<=72h, continuously assessing the risk of cerebral edema and hydrocephalus during hospitalization, and assessing the risk of poor prognosis 90 days before discharge.

[0063] Using the patient's admission time as the baseline zero point (t=0), the collection timestamps of each data point in the multimodal data are converted into offsets Δt relative to the admission time. The formula for calculating the offset Δt is:

[0064] Δt = tcollection - tadmission (1).

[0065] Where tacquisition is the acquisition time of multimodal data, and tadmission is the time when the patient is admitted and registered.

[0066] The collection time for imaging data is based on the examination execution time; the collection time for clinical laboratory data is based on the blood collection or sampling time; the collection time for medical history and physical examination data is based on the assessment execution time; and the collection time for treatment intervention data is based on the start time of the intervention.

[0067] The feature extraction module is used to receive multimodal time-series correlation data sets and extract radiological feature sets, clinical feature sets, and time-series dynamic feature sets from the multimodal time-series correlation data sets.

[0068] like Figure 2 As shown, the feature extraction module includes:

[0069] The image feature extraction unit is used to segment the hematoma region through a three-dimensional convolutional neural network, calculate the hematoma volume, morphological features and density distribution features, and perform differential calculation on image features at different time series to generate dynamic evolution features of the hematoma; the prediction and reasoning module is used to determine the risk level of poor prognosis based on the dynamic evolution features of the hematoma.

[0070] Specifically, in this embodiment, the image feature dimensions include CT image signs (island sign, satellite sign, vortex sign, black hole sign, mixed sign, fluid level), intraventricular hemorrhage volume, cerebral edema volume, and hydrocephalus volume. In this embodiment, 3DSlicer segmentation and Evans index are used to measure cerebral edema volume and hydrocephalus volume. The image feature extraction unit receives head CT image data from each disease stage group in the multimodal temporal correlation data set and uses a segmentation model based on a three-dimensional convolutional neural network (3D-CNN) to automatically segment the hematoma region. The segmentation model adopts an encoder-decoder architecture. The encoder extracts high-dimensional semantic features of the image by downsampling through multiple layers of three-dimensional convolutional layers and pooling layers, and the decoder restores the spatial resolution by upsampling through transposed convolutional layers and skip connections, finally outputting a probability map of the hematoma region with the same resolution as the input image. The probability map is binarized (threshold is 0.5) to obtain a hematoma region mask.

[0071] Based on the hematoma region mask obtained from segmentation, the image feature extraction unit calculates the hematoma volume. The hematoma volume is calculated as follows:

[0072] V = Q × V0 (2).

[0073] Where Q represents the total number of hematoma voxels marked in the hematoma region mask, and V0 represents the physical volume of the voxels in the image (slice thickness × pixel width × pixel height), yielding the hematoma volume value in milliliters (mL). The hematoma volume calculation is performed separately for each disease stage group when multiple CT images exist, outputting the hematoma volume sequence at each time point of that stage.

[0074] like Figure 2 As shown, the feature extraction module also includes:

[0075] The clinical feature extraction unit is used to encode the temporal change trend of biochemical test indicators using a long short-term memory network, extract baseline feature vectors and treatment strategy feature vectors, and then fuse and encode them through a multilayer perceptron to generate a clinical feature set; the prediction and reasoning module is used to determine the risk level of complications based on the clinical feature set.

[0076] Specifically, the indicators within a single disease stage group are arranged chronologically and input into a Long Short-Term Memory (LSTM) network. The LSTM network updates the cell state step-by-step and outputs a hidden state vector through gating mechanisms including input, forget, and output gates. The output hidden state vector of the last time step of the LSTM network is taken as the temporal trend encoding vector for that biochemical indicator within that disease stage group. The temporal trend encoding vectors between different disease stage groups are concatenated in stage order to form a complete trend encoding sequence representing the evolution of each biochemical indicator throughout the entire disease course from admission to discharge.

[0077] The clinical feature extraction unit extracts baseline feature vectors from the admission stage group within the multimodal time-series correlation dataset. The baseline feature vector includes at least the following structured fields:

[0078] Demographic characteristics: name, age, and sex.

[0079] Past medical history characteristics: history of hypertension, history of diabetes, history of atrial fibrillation, history of taking anticoagulants and antiplatelet drugs.

[0080] Clinical assessment features upon admission: systolic blood pressure, diastolic blood pressure, and body temperature.

[0081] Admission time characteristics: the time interval between the onset of illness and admission.

[0082] The clinical feature extraction unit performs minimum and maximum normalization on the numerical features in the baseline feature vector, performs one-hot encoding on the categorical features, and performs integer encoding on the ordered hierarchical features. The normalized and encoded fields are then concatenated in a fixed order to form a standardized baseline feature vector.

[0083] The clinical feature extraction unit extracts treatment intervention data from hospitalization stage groups within the multimodal time-series correlation dataset, generating treatment strategy feature vectors. These feature vectors must include at least the following fields:

[0084] Whether or not minimally invasive surgery was performed: a binary feature, where 1 indicates that minimally invasive surgery was performed and 0 indicates that minimally invasive surgery was not performed.

[0085] Timing of surgery: expressed as the offset from the time of admission, in hours.

[0086] Surgical procedure types include, but are not limited to, stereotactic puncture and drainage, neuroendoscopic hematoma evacuation, and craniotomy hematoma evacuation, represented by unique thermal codes.

[0087] Drug intervention types include, but are not limited to, whether hemostatic drugs are used, whether dehydrating and intracranial pressure-reducing drugs are used, and whether antiepileptic drugs are used. Each drug type is represented by a binary characteristic.

[0088] Timing and duration of drug intervention: expressed as the offset between the first administration time of each drug and the admission time and the total duration of medication (unit: hours).

[0089] At admission (t=0): Initial imaging and biochemical tests are collected, generating Image Report 1 and Risk Report 1 (risk of hematoma expansion); within 6 hours of admission: Repeat imaging and biochemical tests are performed, comparing with the initial images to determine hematoma expansion; within 24 hours of onset: Based on CT findings and clinical data, a comprehensive assessment of hematoma expansion risk is made, generating surgical recommendations; within 72 hours of admission: A decision is made on surgical / conservative treatment, and surgical patients enter the postoperative monitoring phase; during hospitalization: continuous monitoring of cerebral edema, hydrocephalus, and complication risks is conducted; at discharge: a prognostic assessment at discharge and a 90-day adverse prognostic risk report are generated; 90 days after discharge: follow-up data on actual prognosis is collected, and the model is updated accordingly. Different time windows trigger different prediction tasks and report generation.

[0090] like Figure 2 As shown, the feature extraction module also includes:

[0091] The temporal feature extraction unit is used to calculate the slope of numerical change, extreme value difference, and fluctuation entropy of the same feature type across disease stages, generating a temporal dynamic feature set; the prediction and reasoning module is used to determine the risk level of disease progression based on the temporal dynamic feature set.

[0092] Specifically, the temporal feature extraction unit outputs the generated temporal dynamic feature set to the prediction and inference module. Based on the temporal dynamic feature set, and combined with the first slope threshold and the second slope threshold pre-set in the multimodal fusion prediction model, the prediction and inference module comprehensively judges the risk level of the patient's condition worsening.

[0093] In this embodiment, disease progression includes progressive increase in hematoma volume and progressive decrease in hematoma density. The risk level of disease progression includes at least the following three levels:

[0094] Low risk: The slope of the change in hematoma volume characteristics is less than the preset first slope threshold (e.g., ≤0.5mL / stage), the extreme value difference is less than the preset first volume difference threshold (e.g., ≤5mL), and the fluctuation entropy is less than the preset first entropy threshold (e.g., ≤0.3), indicating that the hematoma volume is stable or slowly shrinking and the disease progression is highly regular.

[0095] Medium risk: The slope of the change in hematoma volume characteristics is between the first slope threshold and the preset second slope threshold (e.g., 0.5~2.0 mL / stage), the extreme value difference is between the first volume difference threshold and the preset second volume difference threshold (e.g., 5-15 mL), or the fluctuation entropy is between the first entropy threshold and the preset second entropy threshold (e.g., 0.3-0.6), indicating that the hematoma has a certain degree of dynamic change but is still within a controllable range;

[0096] High risk: The slope of the change in hematoma volume characteristics is greater than or equal to the preset second slope threshold (e.g., ≥2.0mL / stage), the extreme value difference is greater than or equal to the preset second volume difference threshold (e.g., ≥15mL), or the fluctuation entropy is greater than or equal to the preset second entropy threshold (e.g., ≥0.6), indicating that the hematoma volume is rapidly expanding and the direction of disease progression is highly uncertain, suggesting a high-risk state of rapid deterioration of the condition.

[0097] The risk level of disease progression can be further refined into three dimensions: hematoma progression risk level, cerebral edema risk level, and hydrocephalus risk level. Each dimension has three threshold levels: low, medium, and high.

[0098] The risk of hematoma expansion is assessed based on the change in hematoma volume; the risk of cerebral edema is assessed based on the change in edema volume and the degree of midline shift; and the risk of hydrocephalus is assessed based on the change in ventricular volume and the change in the Evans index.

[0099] Doctors can adjust various thresholds based on the patient's actual condition. For example, in this embodiment, the intermediate threshold for the risk of hematoma expansion is an increase in hematoma volume ≥6mL or ≥33%. Therefore, when the patient's hematoma volume increases by ≥6mL or ≥33%, the patient is judged to have an intermediate risk of hematoma expansion. The prediction and inference module receives imaging feature sets, clinical feature sets, and temporal dynamic feature sets. It also constructs a multimodal fusion prediction model using a convolutional neural network model. The multimodal fusion prediction model uses a cross-modal attention mechanism to weightedly fuse the imaging feature sets, clinical feature sets, and temporal dynamic feature sets to obtain a fused feature vector. Based on the fused feature vector, SHAP values ​​are used for feature importance analysis to generate prognostic prediction results. The prognostic prediction results include at least the risk level of poor prognosis, the risk level of disease progression, and the risk level of complications.

[0100] The prediction output module receives the prognostic prediction results, analyzes the corresponding prognostic treatment suggestions based on the prognostic prediction results, generates a comprehensive prognostic report based on the prognostic prediction results and prognostic treatment suggestions, and determines whether the patient has followed the comprehensive prognostic report to perform various prognostic treatments based on the patient's latest multimodal data and the comprehensive prognostic report. Based on the execution status of each prognostic treatment, a task scheduling report is generated.

[0101] like Figure 3 As shown, the prediction output module includes:

[0102] The decision suggestion generation unit is used to input historical prognostic results and their treatment plans, construct a clinical decision rule base based on historical prognostic results and their treatment plans, and generate prognostic suggestions based on prognostic prediction results and the clinical decision rule base. The prognostic suggestions include at least the timing of surgery, complication prevention measures, and rehabilitation management plans.

[0103] Specifically, the decision suggestion generation unit receives the prognostic prediction results and extracts the current patient's feature tags from them. The current patient's feature tags include at least: each risk level in the prognostic prediction results (poor prognostic risk level, disease progression risk level, complication risk level, hematoma volume at admission, hematoma location, age, and past medical history). The decision suggestion generation unit uses the current patient's feature tags as query conditions and performs rule matching in the clinical decision rule base. The matching method is as follows: the matching degree between the current patient's feature tags and the triggering conditions of each rule in the rule base is calculated. The matching degree uses a weighted similarity calculation method, assigning higher weights to features related to prognostic risk levels, and selecting the rule with the highest matching degree as the optimal matching rule. The decision suggestion generation unit generates prognostic suggestions based on the processing plan corresponding to the optimal matching rule.

[0104] like Figure 3 As shown, the prediction output module also includes:

[0105] The task scheduling unit generates an executed list based on the executed pre-processing content and generates an unexecuted reminder report based on the unexecuted pre-processing content.

[0106] Specifically, the task scheduling unit receives the prognostic treatment recommendations from the comprehensive prognostic report and extracts the various prognostic treatment items to be executed. Each prognostic treatment item includes at least: surgical intervention, drug intervention, imaging follow-up, biochemical test follow-up, neurological function assessment, and rehabilitation treatment.

[0107] The task scheduling unit connects with the hospital information system, image archiving and communication system, laboratory information system and electronic medical record system through data interfaces, and queries the actual execution records of each prognosis treatment item according to the preset monitoring cycle (monitoring once every 6 hours).

[0108] The task scheduling unit compares the preset execution plans of each pre-processing item with the actual execution records to identify executed and unexecuted pre-processing items. The criteria for execution are: the item has an actual execution record and the execution time is within the preset execution time window or within a reasonable extension period of no more than 48 hours after the preset execution time window; the criteria for non-execution are: the item has no actual execution record, or the actual execution time exceeds the preset execution time window by more than 48 hours without a reasonable explanation for the extension.

[0109] The data management module is used to store, query, and update comprehensive prognostic reports for patients throughout their entire disease course.

[0110] The feedback verification module is used to collect multimodal data of patients 90 days after actual discharge as the actual prognostic result. It compares the predicted prognostic result with the actual prognostic result to obtain the comparison result, and feeds the comparison result back to the prediction inference module. The prediction inference module is used to update the multimodal fusion prediction model based on the comparison result.

[0111] Specifically, the feedback verification module automatically triggers the data collection process on the 90th day after the patient's discharge (90 days after the discharge date as the baseline). The patient can input multimodal data 90 days after discharge into the feedback verification module through a mobile device (phone, computer, tablet, etc.; in this embodiment, a mobile phone is used). The feedback verification module calculates the overall accuracy of the patient's prediction based on the comparison result matrix. The prediction inference module updates the model parameters of the multimodal fusion prediction model using an online incremental learning method combined with the overall accuracy.

[0112] Example 2:

[0113] Unlike the embodiments described above, as Figure 4As shown, a method for predicting the prognosis of cerebral hemorrhage based on multimodal data fusion includes the following steps:

[0114] S1, Data Acquisition: Acquire the patient's current multimodal data, align the multimodal data to a unified time axis according to the acquisition time, and construct a multimodal time-series correlated data group.

[0115] S2, Feature Extraction: Extracting radiological feature sets, clinical feature sets, and time-series dynamic feature sets from multimodal temporal correlation datasets.

[0116] S3, Multimodal Fusion Prediction: Input the imaging feature set, clinical feature set, and temporal dynamic feature set into the multimodal fusion prediction model.

[0117] S4, Prognostic Management Recommendation Generation: Obtain the comprehensive prognostic report, and output the prognostic prediction results and prognostic management recommendations in the comprehensive prognostic report; In this embodiment, the prognostic management recommendations include at least recommendations for regular follow-up examinations, surgical treatment, and conservative treatment.

[0118] S5, Task Scheduling and Execution Supervision: Collects the patient's current prognostic treatment content and execution status, generates an executed list based on the executed prognostic treatment content, and generates an unexecuted reminder report based on the unexecuted prognostic treatment content.

[0119] S6, Feedback Validation and Model Update: Collect multimodal data of patients 90 days after discharge and feed the multimodal data of patients 90 days after discharge back to the multimodal fusion prediction model to update the multimodal fusion prediction model.

[0120] Specifically, this method constructs a closed-loop management process covering the entire course of cerebral hemorrhage patients from admission to discharge through four steps: data acquisition, feature extraction, result prediction, and prognostic processing.

[0121] This method first uses the admission time as a benchmark to unify and align multi-source heterogeneous data such as imaging, laboratory tests, medical history, and treatment interventions to the same time axis and perform stage aggregation, effectively solving the problem of inconsistent time granularity and difficulty in fusion of clinical multimodal data.

[0122] Based on this, a three-dimensional convolutional neural network was used to accurately segment the hematoma region and extract dynamic evolution features, a long short-term memory network was used to capture the temporal change trend of biochemical indicators, and the dynamic evolution pattern across disease stages was quantified by numerical change slope and fluctuation entropy. This enabled a three-dimensional and quantitative characterization of the patient's condition from three dimensions: imaging, clinical, and temporal dynamics.

[0123] Subsequently, an adaptive weighted fusion of features from different modalities is performed through a cross-modal attention fusion mechanism to fully leverage the complementary advantages of multimodal data and generate accurate risk level predictions.

[0124] Finally, based on the output prediction results, this method further links the prediction results with the clinical decision rule base to automatically match and generate individualized suggestions for surgical timing, complication prevention and rehabilitation management. Through the task scheduling unit, it continuously tracks the actual implementation of prognostic treatment, forming a complete business closed loop of prediction, suggestion, execution and supervision. Compared with the existing technology that only predicts the results, this method improves the completeness and practicality of clinical decision support.

[0125] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A prognostic prediction system for cerebral hemorrhage based on multimodal data fusion, characterized in that, include: The data acquisition and preprocessing module is used to acquire patients' multimodal data and align the multimodal data to a unified time axis according to the acquisition time to construct a multimodal time-series correlated data group; the multimodal data includes at least imaging data, clinical laboratory data, medical history and physical examination data, and treatment intervention data; The key time points in the timeline include at least the time of admission (t=0), within 6 hours of admission, within 24 hours of onset, within 72 hours of admission, the time of discharge, and 90 days after discharge; The feature extraction module is used to receive multimodal time-series correlated data sets and extract imaging feature sets, clinical feature sets, and time-series dynamic feature sets from the multimodal time-series correlated data sets; The prediction and inference module receives imaging feature sets, clinical feature sets, and temporal dynamic feature sets. It also constructs a multimodal fusion prediction model using a convolutional neural network model. This model uses a cross-modal attention mechanism to weightedly fuse the imaging feature sets, clinical feature sets, and temporal dynamic feature sets to obtain a fused feature vector. Based on this fused feature vector, SHAP values ​​are used for feature importance analysis to generate prognostic prediction results. The prediction output module is used to receive the prognosis prediction results, analyze the corresponding prognosis treatment suggestions based on the prognosis prediction results, generate a comprehensive prognosis report based on the prognosis prediction results and prognosis treatment suggestions, and determine whether the patient has followed the comprehensive prognosis report to perform various prognosis treatments based on the patient's latest multimodal data and the comprehensive prognosis report. It also generates a task scheduling report based on the execution status of each prognosis treatment. The data management module is used to store, query, and update comprehensive prognostic reports for patients throughout their entire disease course.

2. The prognostic prediction system for cerebral hemorrhage based on multimodal data fusion according to claim 1, characterized in that, The specific method for aligning the execution time of the data acquisition and preprocessing modules is as follows: Based on the admission time, the acquisition time of multimodal data is converted into an offset relative to the admission time; The multimodal data are divided into corresponding disease stage groups according to the preset time window. The disease stage groups include the admission stage, hospitalization stage, and discharge stage. Time-series aggregation is performed on multiple data points of the same modality within the same disease stage to generate representative feature values ​​for that modality at that stage.

3. The prognostic prediction system for cerebral hemorrhage based on multimodal data fusion according to claim 2, characterized in that, The prognostic prediction results should include at least the risk level of poor prognosis, the risk level of disease progression, and the risk level of complications; the risk level of disease progression includes the risk level of hematoma expansion, the risk level of cerebral edema, and the risk level of hydrocephalus.

4. The prognostic prediction system for cerebral hemorrhage based on multimodal data fusion according to claim 3, characterized in that, The feature extraction module includes: The image feature extraction unit is used to segment the hematoma region through a three-dimensional convolutional neural network, calculate the hematoma volume, morphological features and density distribution features, and perform differential calculation on image features at different time series to generate dynamic evolution features of the hematoma; the prediction and reasoning module is used to determine the risk level of poor prognosis based on the dynamic evolution features of the hematoma.

5. The prognostic prediction system for cerebral hemorrhage based on multimodal data fusion according to claim 4, characterized in that, The feature extraction module also includes: The clinical feature extraction unit is used to encode the temporal change trend of biochemical test indicators using a long short-term memory network, extract baseline feature vectors and treatment strategy feature vectors, and then fuse and encode them through a multilayer perceptron to generate a clinical feature set; the prediction and reasoning module is used to determine the risk level of complications based on the clinical feature set.

6. The prognostic prediction system for cerebral hemorrhage based on multimodal data fusion according to claim 5, characterized in that, The feature extraction module also includes: The temporal feature extraction unit is used to calculate the slope, extreme value difference, and fluctuation entropy of the numerical changes of the same feature type across disease stages, and to obtain data on changes in hematoma volume, edema volume, midline shift, ventricular volume, and Evans index, generating a temporal dynamic feature set. The prediction and inference module is used to determine the risk level of hematoma expansion based on changes in hematoma volume, the risk level of cerebral edema based on changes in edema volume and midline shift, and the risk level of hydrocephalus based on changes in ventricular volume and Evans index. The risk levels of hematoma expansion, cerebral edema, and hydrocephalus are all categorized as high, medium, and low risk.

7. The prognostic prediction system for cerebral hemorrhage based on multimodal data fusion according to claim 6, characterized in that, The prediction output module includes: The decision suggestion generation unit is used to input historical prognostic results and their treatment plans, construct a clinical decision rule base based on historical prognostic results and their treatment plans, and generate prognostic suggestions based on prognostic prediction results and the clinical decision rule base. The prognostic suggestions include at least the timing of surgery, complication prevention measures, and rehabilitation management plans.

8. The prognostic prediction system for cerebral hemorrhage based on multimodal data fusion according to claim 7, characterized in that, The prediction output module also includes: The task scheduling unit generates an executed list based on the executed pre-processing content and generates an unexecuted reminder report based on the unexecuted pre-processing content.

9. The prognostic prediction system for cerebral hemorrhage based on multimodal data fusion according to claim 8, characterized in that, Also includes: The feedback verification module is used to collect multimodal data of patients 90 days after actual discharge as the actual prognostic result, compare the prognostic prediction result with the actual prognostic result, obtain the comparison result, and feed the comparison result back to the prediction inference module. The prediction inference module is used to update the multimodal fusion prediction model based on the comparison results.

10. A method for predicting the prognosis of cerebral hemorrhage based on multimodal data fusion, characterized in that, Includes the following steps: S1, Data Acquisition: Acquire the patient's current multimodal data, align the multimodal data to a unified time axis according to the acquisition time, and construct a multimodal time-series correlated data group; S2, Feature Extraction: Extracting radiological feature sets, clinical feature sets, and time-series dynamic feature sets from multimodal temporal correlation datasets; S3, Multimodal Fusion Prediction: Input the imaging feature set, clinical feature set, and temporal dynamic feature set into the multimodal fusion prediction model; S4, Prognostic Treatment Recommendation Generation: Obtain the comprehensive prognostic report and output the prognostic prediction results and prognostic treatment recommendations from the comprehensive prognostic report; S5, Task Scheduling and Execution Supervision: Collects the patient's current prognostic treatment content and execution status; generates an executed list based on the executed prognostic treatment content; and generates an unexecuted reminder report based on the unexecuted prognostic treatment content. S6, Feedback Validation and Model Update: Collect multimodal data of patients 90 days after discharge and feed the multimodal data of patients 90 days after discharge back to the multimodal fusion prediction model to update the multimodal fusion prediction model.

Images