Method and system for evaluating early risk of heart failure by using multi-modal data

Abstract

The invention provides a method and system for carrying out heart failure early risk assessment by using multi-modal data, and the method comprises the steps: collecting medical data of a patient, and carrying out the preprocessing of the medical data; and inputting the preprocessed medical data into a pre-established heart failure assessment model, performing assessment by using the heart failure assessment model, and outputting an assessment result, the assessment result comprising a heart failure category or a heart failure risk level of the patient. According to the method, the data from different sources are comprehensively analyzed, so that the relevance and complementarity among the data sources are fully mined. By using the BiLSTM-CRF model, the time sequence data of a patient can be effectively analyzed, the dynamic change of the patient can be captured, and the dependency relationship among different characteristics can be established, so that a doctor can be helped to more accurately identify the symptom and risk level of heart failure.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical health, in particular to a method and system for early risk assessment of heart failure using multi-modal data. BACKGROUND

[0002] With the increasing incidence of cardiovascular diseases, heart failure has gradually become a serious public health problem worldwide. Early diagnosis and precise treatment of heart failure are crucial for reducing mortality and improving the quality of life of patients. However, traditional methods of diagnosing heart failure usually rely on single data sources such as clinical symptoms, physical examination, blood marker detection, and imaging examination. Although these methods are widely used in clinical practice, they also have obvious limitations. In the diagnosis of heart failure, artificial intelligence can analyze historical medical records, clinical examination results, imaging data, and other multi-dimensional data to achieve accurate diagnosis and risk assessment. For example, using deep learning models such as convolutional neural networks (CNN) and recurrent neural networks (RNN) to analyze electrocardiogram (ECG) and echocardiogram (Echocardiogram) imaging data can automatically identify and quantify relevant indicators of heart failure, thereby assisting doctors in making an earlier diagnosis.

[0003] However, most current heart failure diagnosis models still mainly rely on single data sources, such as only using imaging examination or single biomarker detection for evaluation. These single-modal diagnostic methods often fail to fully utilize the relevance and complementarity between different data sources. Data from different sources has different characteristics and information potential. For example, imaging data provides spatial information about the structure and function of the heart, while biomarkers reflect biochemical changes in heart failure, and clinical symptoms and signs are often closely related to the patient's clinical status. A single data source cannot fully reflect the patient's pathophysiological process and may miss some important diagnostic clues. SUMMARY

[0004] To solve the above problems, the purpose of the present application is to provide a method and system for early risk assessment of heart failure using multi-modal data, as follows.

[0005] The present application provides a method for early risk assessment of heart failure using multi-modal data, characterized in that the method comprises: Collecting medical data of patients and pre-processing the medical data; the medical data includes electronic health records (EHR) of patients, clinical examination data, electrocardiogram, imaging data such as echocardiogram, and laboratory results; inputting the preprocessed medical data into a pre-established heart failure evaluation model, outputting an evaluation result after evaluation by using the heart failure evaluation model, the evaluation result including a heart failure category or a heart failure risk level of the patient; The heart failure evaluation model uses a bidirectional long short-term memory network to model time series data and capture dynamic characteristics of changes in the patient's state over time, and uses a data analysis or deep learning model to automatically extract features associated with heart failure.

[0006] Optionally, the heart failure evaluation model includes a bidirectional long short-term memory network BiLSTM layer and a conditional random field CRF layer. BiLSTM layer: used to capture forward and backward dependencies in time series data, input data is time series data generated from patient medical data, and output data is time series feature representation extracted by BiLSTM processing; CRF layer: used to model and learn dependencies between features, input data is time series feature representation extracted by the BiLSTM layer, and output data is an evaluated heart failure category or a heart failure risk level.

[0007] Optionally, the model training of the heart failure evaluation model is as follows: Training set: using a patient data set labeled with heart failure diagnosis for training; Loss function: using cross-entropy loss function for classification task optimization; Optimization algorithm: using Adam optimization algorithm to automatically adjust network parameters and minimize training error; Regularization and parameter adjustment: introducing Dropout and L2 regularization techniques to prevent overfitting; adjusting hyperparameters through cross-validation.

[0008] Optionally, the preprocessing of the medical data includes: Analyzing key data related to heart failure evaluation in the medical data, including clinical symptoms, examination indicators, and vital signs; the clinical symptoms include shortness of breath, fatigue, and edema, the examination indicators include echocardiography results and BNP levels in blood, and the vital signs include heart rate, blood pressure, respiratory rate, and oxygen saturation; Data cleaning of the key data to handle missing values and outliers and ensure data integrity; Standardization and normalization of the cleaned key data: ensuring that different types of data are on the same scale to obtain preprocessed medical data.

[0009] The present application provides a system for early risk assessment of heart failure using multi-modal data, characterized in that the system comprises: a data collection module configured to collect and preprocess medical data of a patient, wherein the medical data comprises electronic health records (EHR) of the patient, clinical examination data, electrocardiogram, imaging data such as echocardiogram, and laboratory results; an evaluation module configured to input the preprocessed medical data into a pre-established heart failure evaluation model, and output an evaluation result after evaluation by the heart failure evaluation model, wherein the evaluation result comprises a heart failure category or a heart failure risk level of the patient; The heart failure evaluation model uses a bidirectional long short-term memory network to model time series data and capture dynamic characteristics of the patient state over time, and uses a data analysis or deep learning model to automatically extract features associated with heart failure.

[0010] The application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method for early risk assessment of heart failure by using multi-modal data.

[0011] The application provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the method for early risk assessment of heart failure by using multi-modal data.

[0012] The multi-modal data fusion of the application refers to deep collaboration of different source data through a hierarchical processing mechanism in model learning: the modality feature extraction layer uses BiLSTM to capture the dynamic change rule of blood pressure, heart rate and other indicators for time series data of clinical symptoms; for imaging data (such as echocardiogram), CNN is used to extract spatial structure features such as ventricular wall thickness; for laboratory examination (such as BNP level), a full connection layer is used to convert it into a high-dimensional feature vector. The attention fusion layer learns dynamic weights to strengthen the contribution of biomarkers such as BNP in the acute heart failure scenario, and to improve the weight of imaging structure features in the chronic progression. Finally, the CRF layer models the association rule of "structural abnormalities-biochemical indicators rising-symptom aggravation", and realizes the complementary integration of multi-source information. For example, by combining imaging data with medical history and physical sign information in the clinical medical record, it can more accurately determine whether the patient is at risk of heart failure; by combining biomarker data, it can provide a basis for personalized treatment.

[0013] Based on the method and system for early risk assessment of heart failure using multi-modal data according to the present application, the accuracy and efficiency of heart failure (heart failure) assessment can be improved, especially based on multi-source medical data (such as electronic health records, electrocardiogram, echocardiogram, laboratory test results, etc.), the machine learning and deep learning technology is used to realize the automatic auxiliary assessment. By using the BiLSTM-CRF (Bidirectional Long Short-Term Memory Network and Conditional Random Field) model, the time series data of the patient can be effectively analyzed, the dynamic changes can be captured, and the dependency between different features can be established, so as to help the doctor more accurately identify the symptoms and risk level of heart failure.

[0014] The above and other objects, advantages and features of the present application will become more apparent from the following detailed description of the preferred embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0015] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are intended to illustrate preferred embodiments of the present application, and should not be considered limiting of the present application. Indeed, the present application can vary from these preferred embodiments without departing from its scope, and the scope of the present application should be measured only in accordance with the claims. In the drawings: Figure 1 is a method for early risk assessment of heart failure using multi-modal data according to an embodiment of the present application; Figure 2 is a working schematic diagram of a heart failure assessment model according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] Embodiments of the present application will be described below in conjunction with the accompanying drawings, and those skilled in the art should understand that these embodiments are only used to explain the present application, but are not limiting.

[0017] An embodiment of the present application provides a method for early risk assessment of heart failure using multi-modal data, as shown in Figure 1 The method for early risk assessment of heart failure using multi-modal data according to the present embodiment includes: S1, collecting medical data of a patient and pre-processing the medical data; the medical data includes electronic health records (EHR) of the patient, clinical examination data, electrocardiogram, imaging data such as echocardiogram, and laboratory results.

[0018] Specifically, the pre-processing of the medical data in the present embodiment includes: first, analyzing the key data related to heart failure assessment in the medical data, including the patient's electronic health records (EHR), clinical examination data, electrocardiogram (ECG), imaging data (such as echocardiogram), and laboratory results (such as blood test).

[0019] Secondly, features highly related to heart failure diagnosis are selected through statistical analysis or expert experience. For example, clinical symptoms, examination indexes, vital signs, etc. Among them, the clinical symptoms include shortness of breath, fatigue, edema, the examination indexes include heart ultrasound results, BNP levels in blood, and the vital signs include heart rate, blood pressure, respiratory rate, oxygen saturation; Thirdly, the key data is subjected to data cleaning to process missing values and abnormal values and ensure data integrity. Finally, the key data after cleaning is subjected to standardization and normalization: ensuring that different types of data are on the same scale for model training, so that the preprocessed medical data can be obtained.

[0020] S2, input the preprocessed medical data into a pre-established heart failure evaluation model, use the heart failure evaluation model to output an evaluation result after evaluation, the evaluation result includes a heart failure category or a heart failure risk level of the patient; the heart failure evaluation model uses a bidirectional long short-term memory network to model time series data, capture dynamic features of changes in the patient's state over time, and automatically extract features associated with heart failure using a data analysis or deep learning model.

[0021] That is, considering that the physiological data involved in heart failure diagnosis has time series characteristics (such as changes in heart rate and blood pressure), a BiLSTM (bidirectional long short-term memory network) is used to model time series data to capture dynamic features of changes in the patient's state over time. In feature extraction, features are extracted using a data analysis or deep learning model (such as a convolutional neural network CNN or an autoencoder) to automatically extract useful features, especially continuous signal data such as electrocardiograms.

[0022] In an optional embodiment of the present application, as shown in Figure 2 The heart failure evaluation model includes a bidirectional long short-term memory network BiLSTM layer and a conditional random field CRF layer. BiLSTM layer: to capture the forward and backward dependencies in time series data, BiLSTM can better understand the relevance of data at past and future time steps. The input data is the time series data generated from the patient's medical data, such as the patient's heart rate, blood pressure, respiratory rate, blood test data, etc. as time series input; the output data is the time series feature representation extracted by BiLSTM processing, which is used for further analysis. When BiLSTM learns features, it first processes the input time series data (such as continuous monitoring of heart rate, blood pressure, respiratory rate, and periodic blood test data) by windowing, converting each time series into a fixed-length feature matrix. The forward LSTM unit starts from the beginning of the sequence and learns the dependency between the current heart rate and the previous blood pressure changes (such as the compensatory increase in heart rate after a sudden increase in blood pressure) at each time step. Through the gating mechanism, it filters out key time series segments (such as abnormal fluctuations in respiratory rate at night). The backward LSTM unit, on the other hand, traverses the sequence in reverse from the end to the beginning, capturing the influence of subsequent test data on the current state (such as the increase in BNP level after 3 days as a supplement to the current assessment of cardiac function). The hidden states of the two directions are concatenated at each time step to form a comprehensive time series feature vector that contains "past cause - current state - future trend" at the same time, for example, the progressive relationship "sustained high blood pressure → increased left ventricular load → gradually rising BNP" is encoded as a quantifiable feature representation, providing dynamic time series basis for subsequent multi-modal fusion.

[0023] BiLSTM contains two LSTM layers: one processes the forward sequence (from the beginning to the end of the sequence), and the other processes the reverse sequence (from the end to the beginning of the sequence). The forward LSTM processes data from left to right, capturing the influence of subsequent time steps on the current time step; the reverse LSTM processes data from right to left, capturing the influence of previous time steps on the current time step. The hidden states of the two directions are concatenated together at each time step to form the final hidden state, thus containing both forward and backward context information.

[0024] CRF layer: Conditional random field (CRF) can be used to model and learn the dependencies between features, especially the joint distribution between different symptoms, signs and experimental data. The input data is the time series feature representation extracted by the BiLSTM layer, and the output data is the evaluated heart failure category (such as mild, moderate, severe heart failure or no heart failure) or the risk level of heart failure. CRF can further optimize the features extracted by BiLSTM to ensure the rationality of the prediction results at the sequence level. By using the transition probability between labels, the output of BiLSTM is corrected to make the prediction results more consistent with the actual labeling rules. When the CRF layer generates heart failure prediction results based on the time series feature representation output by BiLSTM, it first maps the time series feature vector to the initial state score at each time (such as the original score of the "no heart failure", "mild heart failure" and other categories at t1 time). Then, by learning the pre-set state transition matrix, the dependency relationship between adjacent states is modeled: for example, when the features at t1 time show "sudden blood pressure rise + heart rate acceleration", CRF will strengthen the probability of transition to "moderate heart failure" at t2 time, while suppressing the unreasonable jump transition from "no heart failure" to "severe heart failure". At the sequence level, CRF searches for the global optimal path by dynamic programming algorithm (such as Viterbi algorithm) to comprehensively consider the state scores and transition probabilities at each time: for example, when the feature sequence extracted by BiLSTM presents a trend of "BNP slowly rising → left ventricular ejection fraction gradually decreasing", CRF will correct the misjudgment at a single time (such as a short period of normal indicators at a certain time), and finally output a sequence label that conforms to the pathological progression rule, i.e. the evolution trajectory of the patient's heart failure category or risk level in the continuous time period, ensuring that the prediction results not only fit the instant features, but also conform to the time sequence logic of heart failure course.

[0025] The model training of the heart failure evaluation model of the present embodiment is as follows: Training set: The MIMIC-III dataset of patients with labeled heart failure diagnosis is used for training.

[0026] Input layer: receives pre-processed multi-modal data, which is divided into time series clinical flow (such as time series of heart rate, blood pressure, etc.), image flow (such as echocardiogram), and static attribute flow (such as laboratory results); Modality feature extraction layer: 1D convolution is used to extract local features for time series clinical flow, deep learning network (ResNet) is used to extract structural features for image flow, and fully connected layer is used to extract features for static attribute flow; Attention fusion layer: self-attention mechanism is used to weight and fuse multi-modal features, dynamically adjusting the weights of each modality; BiLSTM layer: used to capture the forward and backward time series dependency of the fused features, outputting time series feature representation; CRF layer: based on the time series features output by BiLSTM, the state dependency is modeled, and the heart failure category or risk level is output.

[0027] The training process: the model training divides the data set into training, validation and test sets in the ratio of 7:2:1 and formats the input; the parameters are set using the Xavier method during initialization, the BiLSTM forget gate bias is 1, and the hyperparameters are set as batch_size=32, etc.; in the training loop, the forward propagation is processed through each layer to output the prediction sequence, the cross-entropy loss function is used, the Adam optimizer is combined with the gradient clipping to update the parameters through the back propagation; the optimization is performed through Dropout, L2 regularization and early stopping strategy (the F1 score of the validation set is used as the index, and the training is stopped if there is no improvement for 10 consecutive rounds); the 5-fold cross-validation and grid search are used to determine the optimal hyperparameter combination; and the final test set evaluation index is the accuracy 0.89, etc.

[0028] The embodiment of the present application also provides a system for early risk assessment of heart failure by using multi-modal data, and the system comprises: a data collection module configured to collect medical data of a patient and pre-process the medical data; the medical data comprises electronic health records (EHR) of the patient, clinical examination data, electrocardiogram, image data such as echocardiogram, and laboratory results; an evaluation module configured to input the pre-processed medical data into a pre-established heart failure evaluation model, and output an evaluation result after evaluation by using the heart failure evaluation model; the evaluation result comprises a heart failure category or a heart failure risk level of the patient; The heart failure evaluation model uses a bidirectional long short-term memory network to model time series data and capture dynamic features of changes in the patient state over time, and automatically extracts features associated with heart failure by using a data analysis or deep learning model. The specific implementation functions of each module in the embodiment can be referred to the description of the method embodiment.

[0029] The embodiment of the present application also provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the method for early risk assessment of heart failure by using multi-modal data.

[0030] The embodiment of the present application also provides a computer program product comprising a computer program, and the computer program is executed by a processor to implement the steps of the method for early risk assessment of heart failure by using multi-modal data.

[0031] The above only describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solutions falling within the concept of the present application shall be considered as falling within the protection scope of the present application. It should be noted that, for ordinary skilled persons in the technical field, some improvements and refinements without departing from the principles of the present application shall also be considered as falling within the protection scope of the present application.

Claims

1. A method for early risk assessment of heart failure using multimodal data, characterized in that: The method comprises: Collecting and preprocessing the patient's medical data; the medical data includes the patient's electronic health record (EHR), clinical examination data, electrocardiogram (ECG), imaging data such as echocardiogram, and laboratory results; Inputting the pre-processed medical data into a pre-established heart failure assessment model, performing assessment using the heart failure assessment model and outputting an assessment result, the assessment result including the patient's heart failure category or heart failure risk level; The heart failure assessment model uses a bidirectional long short-term memory network to model time series data to capture the dynamic characteristics of the patient's status changing over time, and uses data analysis or a deep learning model to automatically extract features associated with heart failure.

2. The method according to claim 1, characterized in that The heart failure assessment model includes a bidirectional long short-term memory network BiLSTM layer and a conditional random field CRF layer; BiLSTM layer: used to capture forward and backward dependencies in time series data. The input data is the time series data generated by the patient's medical data, and the output data is the time series feature representation extracted by BiLSTM processing; CRF layer: used to model and learn the dependencies between features. The input data is the time series feature representation extracted by the BiLSTM layer, and the output data is the assessed heart failure category or heart failure risk level.

3. The method according to claim 2, characterized in that The model training of the heart failure assessment model is as follows: Training set: Use a patient dataset with heart failure diagnosis annotated for training; Loss function: Cross entropy loss function is used to optimize the classification task; Optimization algorithm: Use the Adam optimization algorithm to automatically adjust network parameters and minimize training errors; Regularization and parameter adjustment: Introduce Dropout and L2 regularization techniques to prevent overfitting; adjust hyperparameters through cross-validation.

4. The method according to claim 1, wherein Preprocessing the medical data includes: Analyze key data related to heart failure assessment in the medical data, including clinical symptoms, examination indicators, and vital signs; the clinical symptoms include shortness of breath, fatigue, and edema; the examination indicators include cardiac ultrasound results and blood BNP levels; and the vital signs include heart rate, blood pressure, respiratory rate, and oxygen saturation; Perform data cleaning on the key data, process missing values ​​and outliers, and ensure data integrity; Standardize and normalize the cleaned key data: ensure that different types of data are on the same scale to obtain preprocessed medical data.

5. A system for early risk assessment of heart failure using multimodal data, characterized in that: The system comprises: A data collection module is used to collect and pre-process the patient's medical data; the medical data includes the patient's electronic health record (EHR), clinical examination data, electrocardiogram (ECG), imaging data such as echocardiogram, and laboratory results; An evaluation module, configured to input the pre-processed medical data into a pre-established heart failure evaluation model, perform evaluation using the heart failure evaluation model, and output an evaluation result, the evaluation result including the patient's heart failure category or heart failure risk level; The heart failure assessment model uses a bidirectional long short-term memory network to model time series data to capture the dynamic characteristics of the patient's status changing over time, and uses data analysis or a deep learning model to automatically extract features associated with heart failure.

6. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method for early risk assessment of heart failure using multimodal data according to any one of claims 1 to 4 are implemented.

7. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method for early risk assessment of heart failure using multimodal data according to any one of claims 1 to 4.

Images