Cardiovascular disease prediction system based on multi-modal fusion

Abstract

The invention relates to the technical field of data classification and prediction, in particular to a cardiovascular disease prediction system based on multi-modal fusion, which comprises an ECG signal processing module; a text processing module; a multi-modal fusion module; the classification module is used for extracting time sequence features of ECG signals through a convolutional neural network, processing semantic information in a medical text report by adopting a large language model and better capturing key features through prompt fine tuning, and the system realizes deep fusion of ECG and text features through a Transform encoder, exerts complementarity of two modals and improves prediction accuracy; meanwhile, the system predicts various cardiovascular diseases by using a multi-label classification model, is suitable for complicated prediction scenes of complications or multiple diseases, and remarkably improves the comprehensiveness and accuracy of cardiovascular disease prediction by introducing a large language model and prompt fine tuning compared with a traditional prediction method of a single ECG signal.

Description

Technical Field

[0001] The present invention relates to the technical field of data classification and prediction, and in particular to a cardiovascular disease prediction system based on multimodal fusion. Background Art

[0002] Cardiovascular diseases (CVDs) are one of the leading causes of death worldwide. According to the World Health Organization (WHO), millions of people die from cardiovascular diseases every year. Due to the aging population and changes in lifestyle, the incidence of cardiovascular diseases has increased year by year, putting tremendous pressure on the global medical system. Early and accurate diagnosis of cardiovascular diseases is crucial to reducing mortality and improving patient prognosis. As an important diagnostic tool, the electrocardiogram (ECG) can provide key information about the health status of the heart by recording the electrical signals generated by the heart every time it beats.

[0003] Traditional machine learning-based ECG classification methods rely on expert knowledge to extract meaningful features, such as RR intervals, P waves, QRS complexes, and T waves. These extracted features are then input into classifiers such as decision trees and support vector machines. However, these methods are highly dependent on the experience and knowledge of experts and have difficulty dealing with complex and large-scale ECG data. With the increase in data volume and the complexity of disease types, the limitations of traditional machine learning-based methods have become increasingly apparent.

[0004] In recent years, deep learning technology has made significant progress in the field of medical imaging and signal processing. The use of neural networks to automatically extract and classify ECG signals has greatly improved diagnostic efficiency and accuracy. In particular, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have shown strong capabilities in processing time series data and extracting complex features. Andrew Ng built a 34-layer deep neural network model based on convolutional neural networks, and classified 12 heart rhythms for 91,232 single-lead ECGs, with an excellent F1 of 0.837. In addition, a 12-lead ECG classification model has been proposed in the prior art. Based on convolutional neural networks and recurrent neural networks, a spatiotemporal attention mechanism is introduced to identify 9 categories.

[0005] Although deep learning performs well in the classification of single ECG signals, in practical applications, single modality information is often insufficient to provide a comprehensive disease analysis. To solve this problem, researchers have begun to explore multimodal methods. Some existing technologies convert the original ECG signal into Gramian angular field (GAF), recursive graph (RP) and Markov transition field (MTF), and then use CNN to extract features and fuse them to achieve classification. Some existing technologies use GADF, PR, and GASF algorithms to convert one-dimensional ECG signals into three different two-dimensional grayscale images, and then use CvT networks to extract features and perform fusion operations. However, most of these methods convert ECG into other domains, and are essentially based on single-modality ECG feature information. In the process of diagnosing cardiovascular diseases, hospitals not only generate ECG data, but also a large number of medical text reports. These medical text reports often involve descriptions of key features of ECGs, which are of great reference value for the accurate diagnosis of diseases. Therefore, combining medical text reports with ECG signals for analysis is expected to significantly improve the accuracy and comprehensiveness of diagnosis.

[0006] In view of this, the present application aims to provide a cardiovascular disease prediction system based on multimodal fusion, combining medical text reports and ECG signals, so as to improve the accuracy and comprehensiveness of predictive diagnosis of cardiovascular diseases. Summary of the invention

[0007] To solve the above problems, the present invention provides a cardiovascular disease prediction system based on multimodal fusion, which is based on an ECG feature extraction module, a medical text report feature extraction module and a multimodal fusion module, and gives full play to the complementary advantages of ECG signals and medical text reports, thereby improving the accuracy and comprehensiveness of the predictive diagnosis of cardiovascular diseases.

[0008] The technical solution adopted by the present invention is:

[0009] A cardiovascular disease prediction system based on multimodal fusion, comprising:

[0010] ECG signal processing module, used to extract electrophysiological features from multi-lead ECG signals;

[0011] The text processing module extracts features from medical text reports based on the large language model GPT-2, and fine-tunes prompts to enhance the model's understanding of medical semantics.

[0012] The multimodal fusion module fuses the features extracted from ECG signals and text reports through the Transformer encoder structure to achieve adaptive weighted fusion of inter-modal features;

[0013] The classification module outputs the prediction results of various cardiovascular diseases through a multi-label classification model, and feeds the information output by LSTM into the full connection for classification.

[0014] Furthermore, the ECG signal processing module adopts a convolutional neural network (CNN) for automatically extracting time series features from multi-lead electrocardiogram signals, and its convolution kernel is a one-dimensional convolution kernel, so as to be suitable for processing ECG sequence data.

[0015] Furthermore, the convolutional neural network is a ResNet18 architecture, which uses a one-dimensional convolutional layer to adapt to the timing of ECG signals and extract local and global features of ECG signals.

[0016] Furthermore, the text processing module extracts features from medical texts through a fine-tuned GPT-2 model, and the fine-tuning adopts a prompt setting to optimize the GPT-2 model's semantic understanding and feature extraction capabilities for texts in medical scenarios.

[0017] Furthermore, the prompt fine-tuning comprises the following steps:

[0018] Introduce the learnable prompt embedding vector into the input layer of GPT-2;

[0019] Fine-tuning on a specific task on a medical text training set allows the prompt and the parameters of subsequent layers of the model to be jointly trained;

[0020] By freezing the remaining parameters of the GPT-2 model, the general language understanding capabilities learned by the model during large-scale pre-training are preserved.

[0021] Furthermore, the multimodal fusion module realizes feature fusion through a Transformer encoder, which includes a multi-head self-attention mechanism and a feedforward neural network for learning the interaction between features of different modalities and dynamically adjusting the feature weights from ECG signals and medical text reports to make the fused features more accurate.

[0022] Furthermore, the classification module uses a multi-label classification model to calculate the probability of existence of each cardiovascular disease through a sigmoid activation function, and outputs a corresponding diagnostic label based on a set threshold to handle the co-occurrence of multiple diseases, and is suitable for the joint diagnosis of multiple cardiovascular diseases.

[0023] Furthermore, the multimodal fusion module also includes a fully connected layer for performing dimension mapping and alignment operations on ECG features and text features before the Transformer encoder.

[0024] Furthermore, the fully connected layer maps the ECG features and the text features to the same hidden dimension to ensure that the features of the two modalities have the same representation space when input into the Transformer encoder, thereby improving the effect of feature fusion.

[0025] Furthermore, it also includes a loss function optimization module, the loss function of which is a binary cross entropy loss, which is used to process multi-label classification to optimize the prediction accuracy of each disease label.

[0026] The beneficial effects of the present invention are as follows:

[0027] The cardiovascular disease prediction system based on multimodal fusion provided by the present invention includes: an ECG signal processing module, which is used to extract electrophysiological features from multi-lead ECG signals; a text processing module, which extracts features of medical text reports based on the large language model GPT-2, and fine-tunes prompts to enhance the model's understanding of medical semantics; a multimodal fusion module, which fuses features extracted from ECG signals and text reports through a Transformer encoder structure to achieve adaptive weighted fusion of inter-modal features; a classification module, which outputs prediction results of multiple cardiovascular diseases through a multi-label classification model, and feeds the information output by LSTM into a fully connected module for classification. The system extracts features through a convolutional neural network. The system uses the temporal characteristics of ECG signals and a large language model to process the semantic information in medical text reports. In the text report processing, the prompt is fine-tuned to enable it to better capture the key features in the medical text. Subsequently, the system uses the Transformer encoder to achieve deep fusion of ECG and text features, giving full play to the complementarity of the two modalities and improving the accuracy of prediction. At the same time, the system uses a multi-label classification model to predict a variety of cardiovascular diseases, which is particularly suitable for complex prediction scenarios of complications or multiple diseases. Compared with traditional prediction methods that only rely on a single ECG signal, the present invention significantly improves the comprehensiveness and accuracy of cardiovascular disease prediction by introducing a large language model and prompt fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is an overall framework diagram of a prediction system in an embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of the structure of a multi-label classification module in an embodiment of the present invention;

[0030] Figure 3 This is a flowchart of text feature extraction of the GPT-2 model in an embodiment of the present invention;

[0031] Figure 4 Schematic diagram of the structure of the Transformer encoder in an embodiment of the present invention;

[0032] Figure 5 is the accuracy of the Super_diag task dimension during the fine-tuning process in the embodiment of the present invention;

[0033] Figure 6 It is a loss function curve of the Super_diag task dimension during the fine-tuning process in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to facilitate the understanding of the present invention, the present invention will be described more comprehensively by way of examples, and preferred embodiments of the present invention are given below. However, the present invention can be implemented in a variety of different forms and is not limited to the embodiments described herein. Any other implementation schemes obtained by modifying or equivalently replacing the technical scheme of the present invention without creative results are within the protection scope of the present invention.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for describing specific embodiments and are not intended to limit the present invention.

[0036] The numerical values ​​disclosed in the embodiments of the present invention are approximate values, not definite values. When the error or experimental conditions permit, all values ​​within the error range may be included without being limited to the specific numerical values ​​disclosed in the embodiments of the present invention.

[0037] like Figure 1 to Figure 6 As shown, this embodiment provides a cardiovascular disease prediction system based on multimodal fusion, including:

[0038] ECG signal processing module, used to extract electrophysiological features from multi-lead ECG signals;

[0039] Specifically, the ECG signal processing module is responsible for receiving the original electrocardiogram data and automatically extracting the time series features therein through a convolutional neural network (CNN). The traditional ECG signal processing method relies on manual feature extraction, while the present invention uses a convolutional neural network with a ResNet18 architecture to automatically learn the high-dimensional features in the ECG signal. In order to adapt to the time series nature of the ECG signal, the convolution layer in the network is changed to a one-dimensional convolution layer;

[0040] The specific implementation steps are as follows:

[0041] Input: The input ECG signal is multi-lead data of size (N, C, T), where N is the number of samples, C is the number of leads, and T is the time step.

[0042] Feature extraction: The timing and electrophysiological features of ECG signals are extracted through one-dimensional convolutional layers and residual network structures.

[0043] Output: The extracted feature matrix size is (N, F), where F is the feature dimension.

[0044] This module can capture the details of the heart's electrical activity, such as key signal features such as QRS complexes, P waves and T waves, and extract global rhythm patterns.

[0045] The medical text processing module extracts features from medical text reports based on the large language model GPT-2, and fine-tunes prompts to enhance the model's understanding of medical semantics.

[0046] Specifically, the medical text processing module uses the pre-trained large language model GPT-2 to extract features from medical text reports. In order to adapt to the specific context of medical text, the system fine-tunes the GPT-2 model with prompts. Prompt fine-tuning allows the model to focus on semantic features unique to the medical field while retaining its pre-trained knowledge;

[0047] The specific implementation steps are as follows:

[0048] Text embedding: First, the input text report is tokenized and converted into word vectors;

[0049] Prompt fine-tuning: Add a learnable prompt vector before the input word embedding, and optimize the model's understanding of medical scenarios through fine-tuning training on medical data;

[0050] Output: The GPT-2 model extracts semantic features from the text, and the output text features are feature matrices of size (N, T), where T is the text feature dimension;

[0051] This module can effectively extract key information such as condition description, symptom analysis, and doctor's diagnosis opinions involved in medical reports.

[0052] The multimodal fusion module fuses the features extracted from ECG signals and text reports through the Transformer encoder structure to achieve adaptive weighted fusion of inter-modal features;

[0053] The multimodal fusion module deeply fuses ECG features and text features through the Transformer encoder. The multi-head self-attention mechanism of the Transformer encoder can adaptively allocate the weights of each modality feature to ensure that key information is fully utilized in the fusion;

[0054] Specific implementation steps:

[0055] Feature alignment: First, the features from the ECG signal and the text report are dimensionally mapped so that the two features are fused in the same representation space;

[0056] Feature fusion: Use the Transformer encoder to perform weighted fusion of the features of the two modalities. The encoder's multi-head attention mechanism can effectively capture the correlation between different modalities.

[0057] Output: The fused features are represented as (N, F+T), where F is the ECG feature dimension and T is the text feature dimension.

[0058] The classification module outputs the prediction results of various cardiovascular diseases through a multi-label classification model, and feeds the information output by LSTM into the full connection for classification;

[0059] Specifically, the classification module is used to predict the probability of the presence of multiple cardiovascular diseases. This module uses a multi-label classification model to perform independent binary classification on each disease category, uses a sigmoid activation function to output the probability of each disease, and determines whether the disease exists based on the set threshold;

[0060] Specific implementation steps:

[0061] Input: feature matrix after multimodal fusion (N, F+T);

[0062] Classification: Predict each cardiovascular disease through the fully connected layer and sigmoid activation function;

[0063] Output: The predicted probability of each disease is (N, K), where K is the number of disease types;

[0064] This module is capable of handling complex diagnostic scenarios with multiple concurrent cardiovascular diseases and is suitable for situations where multiple diseases coexist in clinical practice.

[0065] Specifically, the training and optimization process of the above system uses the Binary Cross-Entropy (BCE) loss function, which is suitable for multi-label classification tasks. The GPT-2 model of the text processing module is frozen during the training process, and only the parameters of the prompt and ECG feature extraction modules are updated, thereby effectively utilizing the knowledge of the pre-trained model.

[0066] During the experiment, the system used large-scale ECG and medical text data from the PTB-XL dataset, which can be divided into different dimensions: Super-diag, Form, Rhythm, Subdiagnosis, Diagnosis, and All. Each dimension contains different types of diseases.

[0067] For details, please refer to Table 1 below, which shows the performance of each category of the PTB-XL dataset in the Super_diag dimension; please refer to Table 2, which shows the average performance of all disease categories in the PTB-XL dataset in each dimension task.

[0068] Table 1

[0069]

[0070]

[0071] Table 2

[0072]

[0073]

[0074] From the above experimental results, it can be seen that the multimodal system combining ECG signals and text reports provided in this embodiment is significantly better than the single-modal ECG signal classification model in the classification accuracy of cardiovascular diseases. The average accuracy of the system reached 91.09%, and the maximum F1 value was 84.88%, showing excellent diagnostic prediction performance.

[0075] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A cardiovascular disease prediction system based on multimodal fusion, characterized in that: include: ECG signal processing module, used to extract electrophysiological features from multi-lead ECG signals; The text processing module extracts features from medical text reports based on the large language model GPT-2, and fine-tunes prompts to enhance the model's understanding of medical semantics. The multimodal fusion module fuses the features extracted from ECG signals and text reports through the Transformer encoder structure to achieve adaptive weighted fusion of inter-modal features; The classification module outputs the prediction results of various cardiovascular diseases through a multi-label classification model, and feeds the information output by LSTM into the full connection for classification.

2. The cardiovascular disease prediction system based on multimodal fusion according to claim 1, characterized in that: The ECG signal processing module adopts a convolutional neural network to automatically extract time series features from multi-lead electrocardiogram signals, and its convolution kernel is a one-dimensional convolution kernel, which is suitable for processing ECG sequence data.

3. The cardiovascular disease prediction system based on multimodal fusion according to claim 2, characterized in that: The convolutional neural network is a ResNet18 architecture, which uses a one-dimensional convolutional layer to adapt to the timing of ECG signals and extract local and global features of ECG signals.

4. The cardiovascular disease prediction system based on multimodal fusion according to claim 3, characterized in that: The text processing module extracts features from medical texts through a fine-tuned GPT-2 model, and the fine-tuning adopts prompt settings to optimize the GPT-2 model's semantic understanding and feature extraction capabilities for texts in medical scenarios.

5. The cardiovascular disease prediction system based on multimodal fusion according to claim 4, characterized in that: The prompt fine-tuning includes the following steps: Introduce the learnable prompt embedding vector into the input layer of GPT-2; Fine-tuning on a specific task on a medical text training set allows the prompt and the parameters of subsequent layers of the model to be jointly trained; By freezing the remaining parameters of the GPT-2 model, the general language understanding capabilities learned by the model during large-scale pre-training are preserved.

6. The cardiovascular disease prediction system based on multimodal fusion according to claim 1, characterized in that: The multimodal fusion module realizes feature fusion through a Transformer encoder, which includes a multi-head self-attention mechanism and a feedforward neural network for learning the interactive relationship between features of different modalities and dynamically adjusting the feature weights from ECG signals and medical text reports to make the fused features more accurate.

7. The cardiovascular disease prediction system based on multimodal fusion according to claim 1, characterized in that: The classification module uses a multi-label classification model to calculate the probability of existence of each cardiovascular disease through a sigmoid activation function, and outputs a corresponding diagnostic label based on a set threshold to handle the situation where multiple diseases coexist, and is suitable for the joint diagnosis of multiple cardiovascular diseases.

8. The cardiovascular disease prediction system based on multimodal fusion according to claim 1, characterized in that: The multimodal fusion module also includes a fully connected layer for performing dimension mapping and alignment operations on ECG features and text features before the Transformer encoder.

9. The cardiovascular disease prediction system based on multimodal fusion according to claim 8, characterized in that: The fully connected layer maps the ECG features and the text features to the same hidden dimension to ensure that the features of the two modalities have the same representation space when input into the Transformer encoder, thereby improving the effect of feature fusion.

10. The cardiovascular disease prediction system based on multimodal fusion according to claim 1, characterized in that: It also includes a loss function optimization module, wherein the loss function of the loss function optimization module is a binary cross entropy loss, which is used to process multi-label classification to optimize the prediction accuracy of each disease label.

Images