Electrocardiogram data analysis method and device based on multi-modal fusion, medium and equipment

Abstract

The invention provides an electrocardiogram data analysis method, device, medium and equipment based on multi-modal fusion, which are applied to early warning of acute coronary syndrome (ACS). The method comprises the following steps: acquiring time sequence data of electrocardiosignals and ballistocardiogram signals as well as structural data such as heart rate, respiratory rate and oxyhemoglobin saturation by using a portable device, and preprocessing the data; constructing a double-branch global and local interactive feature extraction model, respectively extracting local and global features of the electrocardiosignal and the ballistocardiogram signal, and simultaneously bridging the isomerism of the two signal types; after feature mapping is carried out on the structural data, inputting the structural data and features of the time series data into a feature fusion model; and finally inputting the fused features into a disease early warning model, outputting an ACS risk probability, and comparing the ACS risk probability with a preset threshold value to realize real-time monitoring and early warning. The method is of great significance to prevention of ACS and guarantee of life health of high-risk groups with cardiovascular and cerebrovascular diseases.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data processing, and particularly to an electrocardiogram data analysis method, device, medium and equipment based on multi-modal fusion. Background Art

[0002] Acute Coronary Syndrome (ACS) is a serious cardiovascular disease caused by a sudden reduction or interruption of coronary blood flow, including unstable angina, acute myocardial infarction and sudden death. The occurrence of ACS is usually sudden and difficult to predict. Timely and reliable early warning and intervention are crucial for reducing the fatality rate and disability rate.

[0003] With the development of medical technology and the improvement of residents' health awareness, home monitoring has gradually become an important health monitoring method. Home monitoring can not only reduce medical costs and relieve the burden on medical institutions, but also has significant advantages in the management of chronic diseases and sudden diseases. However, there are still many deficiencies in the accuracy, sensitivity and real-time performance of existing home monitoring devices and methods. For example, using contact electrocardiogram monitoring devices during sleep can easily affect sleep quality; there are many limitations and interferences in collecting physiological signals in the sleep scenario, resulting in inaccurate data quality; long-term skin attachment is prone to dermatitis, etc., making it difficult to meet the needs of early warning of ACS. Therefore, there is an urgent need for a monitoring and warning method that can achieve full-process and non-intrusive monitoring of high-risk populations of cardiovascular and cerebrovascular diseases in the home scenario.

[0004] Nowadays, an intelligent assisted diagnosis method that combines deep learning methods represented by multi-modal fusion technology with clinical pain points has become the current research trend. The present invention designs a multi-modal fusion technology to fuse and analyze the time-series data and structural text data collected during the monitoring process for use in realizing early warning of ACS. For the two types of time-series data of electrocardiogram signals and ballistocardiogram signals, a time-series feature extraction model with global and local interaction is designed. The convolutional neural network is used to extract local features, and the self-attention feature of Transformer is used to capture the long-term dependence relationship between signals for extracting global features. The ballistocardiogram signals obtained by using existing portable physiological information monitoring devices are used to reconstruct and preprocess structural data such as heart rate, respiratory rate, and blood oxygen saturation, and feature vectors are obtained by encoding one by one and fused with the time-series feature vectors. Finally, the fused feature vectors are input into the ACS prediction module to calculate the risk probability. The multi-modal fusion and analysis method adopted by the present invention can be applied to construct a risk monitoring and early warning model for ACS, effectively protect the health of high-risk populations of cardiovascular and cerebrovascular diseases, and relieve the pressure on clinical emergency doctors to a certain extent through early detection, early intervention and early treatment. Summary of the Invention

[0005] In view of this, the present invention provides a method, device, medium and equipment for analyzing electrocardiogram data based on multimodal fusion, which can use deep learning algorithms to process and model multimodal data to achieve real-time monitoring and early warning of ACS, effectively protect the physical health of high-risk groups of cardiovascular and cerebrovascular diseases, and relieve the pressure on clinical emergency doctors.

[0006] In a first aspect, the present invention provides a method for analyzing electrocardiogram data based on multimodal fusion, including:

[0007] Obtain multimodal data including time series data and structural data;

[0008] Preprocess the multimodal data;

[0009] Extract local features and global features from the time series data respectively, and fuse the extracted local features and global features to obtain a first feature vector;

[0010] Fuse the structural data to obtain a second feature vector;

[0011] Fuse the first feature vector and the second feature vector to obtain a prediction vector;

[0012] Determine the risk probability based on the prediction vector, and compare it with the risk threshold established by historical research. When the risk probability exceeds the threshold, generate a warning message.

[0013] In one embodiment, obtaining multimodal data including time series data and structural data includes: collecting time series data including electrocardiogram signals and cardiac impact signals, and extracting at least one piece of structural data including heart rate, respiratory rate, and blood oxygen saturation according to the cardiac impact signals; and collecting at least one piece of structural data including age, gender, BMI index, presence or absence of a history of cardiovascular and cerebrovascular diseases, presence or absence of smoking, and presence or absence of alcohol consumption.

[0014] In one embodiment, preprocessing the multimodal data includes: dividing the time series data into heartbeats, and performing standardization processing on the divided time series data using the Z-score method; performing standardization processing on continuous variables in the structural data using the Z-score method, and performing one-hot encoding on discrete variables in the structural data for standardization processing.

[0015] In one embodiment, extracting local features and global features from the time series data respectively includes: inputting the time series data into a multi-layer convolutional neural network model for processing to obtain local features; and inputting the time series data into a multi-layer Transformer model for processing to obtain global features.

[0016] In one embodiment, the convolutional neural network model includes a number of convolutional blocks, and each convolutional block includes a convolutional layer, a normalization layer, an activation function layer, and a pooling layer;

[0017] The convolutional layer is used to extract local features;

[0018] The normalization layer is used to stabilize the training process of the model;

[0019] The activation function layer is used to capture non-linear features;

[0020] The pooling layer is used to reduce the feature dimension.

[0021] In one embodiment, the Transformer model includes a position encoding layer, a multi-head attention layer, a feed-forward layer, a residual connection layer, and a normalization layer;

[0022] The position encoding layer is used to encode the positions of the data;

[0023] The multi-head attention layer is used to extract different features;

[0024] The feed-forward layer is used to map the feature dimension;

[0025] The residual connection layer is used to improve the convergence efficiency and training stability;

[0026] The normalization layer is used to improve the training robustness and convergence speed.

[0027] In one embodiment, a single-layer perceptron is used to extract high-dimensional features of time-series data to obtain a first feature vector; and, a multi-layer perceptron is used to extract high-dimensional features of structural data to obtain a second feature vector.

[0028] In a second aspect, the present invention provides a multi-modal fusion-based electrocardiogram data analysis device, and the device includes:

[0029] A data acquisition module, configured to acquire multi-modal data, including time-series data and structural data;

[0030] A data processing module, configured to preprocess the multi-modal data;

[0031] A feature extraction module, configured to respectively extract local features and global features from the time-series data, and perform feature fusion on the extracted local features and global features to obtain a first feature vector, and perform feature fusion on the structural data to obtain a second feature vector;

[0032] A feature fusion module, configured to fuse the first feature vector and the second feature vector to obtain a prediction vector;

[0033] A risk prediction module, which is used to determine a risk probability based on a prediction vector, compare it with a risk threshold established by historical research, and generate a warning message when the risk probability exceeds the threshold.

[0034] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in any one of the method embodiments in the first aspect are implemented.

[0035] In a fourth aspect, the present invention provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps in any one of the method embodiments in the first aspect are implemented.

[0036] The present invention provides a method, apparatus, electronic device, and storage medium for electrocardiogram data analysis based on multi-modal fusion, which constructs a feature extraction model with dual-branch global and local interaction, extracts local and global features from electrocardiogram signals and ballistocardiogram signal time series data respectively, and at the same time bridges the heterogeneity between the two signal types; performs corresponding feature mapping after preprocessing the ballistocardiogram signals and structural data such as heart rate, respiratory rate, and blood oxygen saturation extracted by existing portable physiological information monitoring devices; inputs the features extracted from the time series data and structural data into a feature fusion model, and inputs the fused features into a disease warning model, and finally outputs the risk probability of acute coronary syndrome. According to the threshold set by prior research, it is judged whether the user wearing it has the risk of suffering from acute coronary syndrome, so as to realize the real-time monitoring and warning of acute coronary syndrome; uses artificial intelligence methods to solve the pain points of inconvenient home monitoring and warning for high-risk groups of cardiovascular and cerebrovascular diseases, realizes full-process and non-intrusive monitoring and warning for users, makes up for the gap in night monitoring, improves the accuracy and real-time performance of monitoring, improves the convenience of home monitoring, helps reduce medical costs, and protects the physical health of residents, and has important social significance and economic value. Description of the Drawings

[0037] Figure 1 is a flowchart of a method for electrocardiogram data analysis based on multi-modal fusion according to an embodiment of the present invention;

[0038] Figure 2 is a method for extracting time series features based on global and local interaction according to an embodiment of the present invention;

[0039] Figure 3 is a structural diagram of an apparatus for electrocardiogram data analysis based on multi-modal fusion according to an embodiment of the present invention. Detailed Embodiments

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention, rather than limiting the present invention. In addition, it should be noted that for the sake of description, only the parts related to the present invention rather than all the structures are shown in the drawings.

[0041] Embodiment 1

[0042] Figure 1 An exemplary flowchart of an electrocardiogram data analysis method based on multimodal fusion according to some embodiments of the present invention is shown. The electrocardiogram data analysis method may include operations S101 to S106.

[0043] It should be understood that although Figure 1 the steps in the flowchart are shown in the order indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, Figure 1 at least a part of the steps in

[0044] may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0045] In the embodiments of the present disclosure, the time-series data may include at least one of an electrocardiogram signal (ECG) and a ballistocardiogram signal (BCG).

[0046] In the embodiments of the present disclosure, the structural data may include one or several of heart rate, respiratory rate, blood oxygen saturation, age, gender, BMI index, presence or absence of a history of cardiovascular and cerebrovascular diseases, presence or absence of smoking, and presence or absence of drinking.

[0047] It can be understood that the time-series data refers to a data column recorded in chronological order, while the structural data is index data independent of time.

[0048] Optionally, one or several items of structural data such as heart rate, respiratory rate, and blood oxygen saturation can be obtained by analyzing the ballistocardiogram signal using a signal analysis method.

[0049] In a possible embodiment, data such as electrocardiogram signals, ballistocardiogram signals, heart rate, respiratory rate, blood oxygen saturation, age, gender, and BMI index can be collected and extracted using existing portable physiological information monitoring devices. Due to their miniaturization, simple operation, low cost, and the ability to achieve real-time collection and multi-channel monitoring, portable devices can be widely used in various scenarios such as homes, hospitals, and nursing homes to meet the health monitoring needs of different populations.

[0050] In operation S102, preprocess the above multi-modal data.

[0051] In the embodiments of the present disclosure, preprocessing the multi-modal data includes: dividing the time series data into heartbeats and performing standardization processing on the divided time series data using the Z-score method; and, performing standardization processing on the continuous variables in the structural data using the Z-score method, and / or performing one-hot encoding on the discrete variables in the structural data for standardization processing.

[0052] For example, the way to divide the time series data into heartbeats can be: first determine the position of the R peak in the signal. Since the duration of each heartbeat is about 0.8 seconds, taking the R peak as the reference point, 0.4 seconds before and after it is taken as one heartbeat.

[0053] It can be understood that dividing the time series data into the above heartbeats can make data processing more accurate to solve the problem that a piece of data contains different heartbeat rhythms and causes ambiguity in the processing model.

[0054] For example, when preprocessing the time series data and the continuous variables in the structural data, using the Z-score method to standardize the above data can be expressed as a formula:

[0055]

[0056] where μ is the mean of the data sample and θ is the standard deviation of the data sample.

[0057] It can be understood that after completing the Z-score standardization, the means of the time series data and the continuous variables in the structural data are both 0, and the standard deviation is 1.

[0058] For example, when processing the discrete variables in the structural data using one-hot encoding, assuming there are n categories, a category c i 's one-hot encoding is represented as a vector v of length n, where v j is the jth element, then it can be expressed as a formula:

[0059]

[0060] where i is the category c iIndex in all categories, where 1 ≤ i ≤ n, and j is the index in vector v, where 1 ≤ j ≤ n.

[0061] It can be understood that using one-hot encoding to process discrete variables in structured data can convert these discrete variables into numerical vectors so that the model can understand and process this data.

[0062] As Figure 2 shown, in operation S103, local features and global features are respectively extracted from the above-mentioned time-series data, and the extracted local features and global features are fused to obtain a first feature vector.

[0063] In one embodiment, the time-series data can be input into a multi-layer convolutional neural network model for processing to obtain local features.

[0064] For example, inputting the preprocessed and standardized time-series data into a convolutional neural network model for local feature extraction, the calculation formula of the model can be expressed as:

[0065]

[0066] Where respectively represent the feature maps of the outputs of the l-th convolutional block and the (l - 1)-th convolutional block, w ik represents the weight kernel between the i-th and k-th neurons, b k represents the bias of the k-th neuron. σ(·) represents the ReLu activation function, and M k represents the activation range of the convolutional kernel.

[0067] For example, the convolutional neural network model can include several convolutional blocks, and each convolutional block can be composed of a convolutional layer, a batch normalization layer, an activation function layer, and a pooling layer. In the convolutional block, the output of the convolutional layer first passes through the batch normalization layer, which can ensure that the mean of the output features is close to 0 and the standard deviation is close to 1, thus stabilizing the training process of the model and improving the training efficiency; subsequently, the activation function layer performs a non-linear mapping on the neurons of the convolutional layer to capture non-linear features; finally, the pooling layer is used to reduce the feature dimension, thereby accelerating training and reducing the computational overhead of the model. Finally, the output Z local of the local features is obtained.

[0068] It can be understood that in the above convolutional neural network model, the first convolutional block is used to process the input signal, and the next convolutional block takes the feature map of the previous convolutional block as input and continues to process it; since more blocks need to focus on higher-order prediction information, as the depth of the convolutional neural network increases, the size of the convolutional kernel and the number of filters will gradually increase; therefore, in order to reduce the computational complexity of the model, average pooling layers are used in the first three convolutional blocks to reduce the size of the features.

[0069] In one embodiment, the preprocessed and normalized time-series data can be input into the Transformer model for global feature extraction.

[0070] For example, the Transformer model can consist of a positional encoding layer, a multi-head attention layer, a feed-forward layer, as well as residual connection layers and normalization layers.

[0071] For example, the calculation formula of positional encoding can be expressed as:

[0072]

[0073] where pos represents the position, the introduction of i is to distinguish odd and even, and d model represents the dimension. Through this kind of positional encoding, it can help the model learn relative positions more easily. F ecg is the ECG feature map after positional encoding.

[0074] It can be understood that in the Transformer model, its core feature is the global feature extraction ability; however, due to the lack of explicit sequence information, positional encoding needs to be introduced so that the model can perceive the order of the input data. Therefore, the positional encoding layer preserves the sequential characteristics of the time-series data by introducing position information into the input vector, and the size of the positional encoding must be consistent with the size of the input data.

[0075] For example, the above-obtained F ecg can be input into the multi-head attention layer for the calculation of scaled dot-product attention, which can be expressed as the formula:

[0076]

[0077] where are respectively mapped from the ECG feature maps after positional encoding and L ecg is the length of the ECG feature map. For the dot-product operation of query and key, it is scaled according to the coefficient for gradient balance after the Softmax operation. For multi-head attention, query, key, and value are respectively linearly projected into and

[0078] Then, the outputs of all heads are concatenated and input into a linear layer to obtain an output, which can be expressed by the formula:

[0079] Z mha = MultiHead(Q, K, V) = Concat(head1, …, head h )

[0080] where head i = Attention(QW Q , KW K , VW V ).

[0081] It can be understood that the multi-head attention layer realizes diverse feature expressions and more comprehensive information capture by enabling each attention head to focus on different feature subspaces of the input data, and it is a key component of the Transformer.

[0082] For example, in the residual connection layer, the output Z mha of the multi-head attention can be added to the initial feature map F ecg of the ECG to generate the final output and input it into the normalization layer. The calculation formula can be expressed as:

[0083] Z = LayerNorm(Z mha + F ecg )

[0084] For example, the feed-forward layer can be composed of two linear transformations and a ReLu activation function, and its calculation formula can be expressed as:

[0085] Z ffn = FFN(Z) = max(0, ZW1 + b1)W2 + b2

[0086] where W1 and W2 represent weights, and b1 and b2 represent biases.

[0087] It can be understood that the feed-forward layer provides efficient feature representations for subsequent tasks through non-linear mapping and dimensional transformation, enhancing the generalization ability of the model.

[0088] For example, through the residual connection layer, Z ffn can be added to Z and passed through a linear normalization layer to obtain the final global feature vector Z ... general , and the calculation formula can be expressed as:

[0089] Z general = LayerNorm(Z ffn + Z)

[0090] It is understandable that the residual connection layer greatly alleviates the vanishing gradient and exploding gradient problems in deep neural networks by introducing skip connections between adjacent layers, while improving the convergence efficiency and training stability of the model; the normalization layer normalizes the output of each layer, making the mean of the output close to 0 and the standard deviation close to 1, thereby effectively improving the robustness and convergence speed of the training process.

[0091] In the embodiments of the present disclosure, the extracted local features and global features can be fused through a single-layer perceptron to obtain the final temporal feature vector, that is, the first feature vector, and its calculation formula can be expressed as:

[0092] Z output = Concat(Z local , Z general )

[0093] Among them, Concat(·) is a feature concatenation operation.

[0094] In this solution, by adopting a global feature extraction model based on Transformer, it can model the global dependencies of the input signal at one time. In contrast, the convolutional neural network model needs to gradually expand the convolutional kernel through multiple layers to achieve a similar effect; and the self-attention mechanism of the Transformer model can dynamically focus on different positions of the input sequence and can flexibly adapt to the changes in the global patterns in the signal, especially suitable for processing long-time sequence data. Compared with the local perception ability of the convolutional neural network model, the Transformer model can more comprehensively reflect the overall trend of the signal when extracting global features.

[0095] In operation S104, feature fusion is performed on the structure data to obtain a second feature vector.

[0096] For example, the way to perform feature fusion on the structure data can be: fusing the features of all encoded structural data and then using a single layer of feature mapping layer to obtain the final high-dimensional vector, and its calculation formula can be expressed as:

[0097]

[0098] y vfinal = f MLP (Concat(y1, y2,... y n ))

[0099] Among them, x i represents a piece of data in the structural data, and a multi-layer perceptron (MLP, MultilayerPerceptron) is used, that is, f MLP(·) Perform high-dimensional mapping of execution features, Concat(·) is a feature concatenation operation, y vfinal represents the finally obtained feature vector, i.e., the second feature vector.

[0100] For example, a multi-layer perceptron (MLP) can include an input layer, a hidden layer, and an output layer. The input layer receives the input of the original data, and each input feature corresponds to a neuron; the hidden layer is usually composed of one or more hidden layers, and the neurons in each layer are connected to the previous layer through an activation function. The neurons in the hidden layer are responsible for extracting useful features from the input data; the output layer performs mapping according to the desired dimension.

[0101] It can be understood that in the foregoing operation S103, when performing feature encoding on the time series data including the electrocardiogram signal and the ballistocardiogram signal, in order to more clearly describe the fusion operation performed with the structural data, the electrocardiogram signal and the ballistocardiogram signal encoding parts can be represented as f encoder , the electrocardiogram signal is represented as x ecg , the ballistocardiogram signal is represented as x bcg , so its calculation formula can also be expressed as:

[0102] y sfinal = f encoder (x ecg , x bcg )

[0103] where y sfinal represents the extracted time series feature vector, i.e., the first feature vector.

[0104] In operation S105, the above first feature vector and second feature vector are fused to obtain a prediction vector.

[0105] For example, the two different modality feature vectors of the first feature vector and the second feature vector are further fused to obtain the final prediction vector y final , and its calculation formula can be expressed as:

[0106] y final = Concat(y vfinal , y sfinal )

[0107] where y vfinal represents the extracted structural data feature.

[0108] In operation S106, based on the prediction vector, a risk probability is determined and compared with the risk threshold established by historical research. When the risk probability exceeds the threshold, a warning message is generated.

[0109] For example, the way to determine the risk probability based on the prediction vector can be: the fused feature vector yfinal It is input into a trained decoder composed of a multi-layer perceptron (MLP), and the decoder outputs the ACS risk probability.

[0110] For example, a multi-layer perceptron (MLP) may include an input layer, a hidden layer, and an output layer. The input layer receives the input of the original data, and each input feature corresponds to a neuron. The hidden layer is usually composed of one or more hidden layers, and the neurons in each layer are connected to the previous layer through an activation function. The neurons in the hidden layer are responsible for extracting useful features from the input data, and the output layer performs mapping according to the desired dimension. When making the final probability prediction, the dimension of the output layer can be changed to 1, so that a value will be output, and then the sigmoid activation function can be used to limit the output value within the range of (0, 1), which is the required risk probability value.

[0111] Embodiment 2

[0112] Refer to Figure 3 , which is a schematic structural diagram of an electrocardiogram data analysis device based on multi-modal fusion provided by an embodiment of the present invention. The device may include: a data acquisition module, a data processing module, a feature extraction module, a feature fusion module, and an ACS risk prediction module, which are specifically described as follows:

[0113] The data acquisition module is used to acquire multi-modal data including time-series data and structural data.

[0114] The data processing module is used to preprocess the multi-modal data.

[0115] The feature extraction module is used to respectively extract local features and global features from the time-series data, and perform feature fusion on the extracted local features and global features to obtain a first feature vector.

[0116] The feature fusion module is used to perform feature fusion on the structural data to obtain a second feature vector; and is used to fuse the first feature vector and the second feature vector to obtain a prediction vector. [[ID=2)]]

[0117] The ACS risk prediction module is used to determine the ACS risk probability based on the prediction vector, and compare it with the risk threshold established by historical research. When the risk probability exceeds the threshold, a warning message is generated.

[0118] In some embodiments, the method for determining the ACS risk probability based on the prediction vector is: input the fused feature vector y final into a trained decoder composed of a multi-layer perceptron (MLP), and the decoder outputs the ACS risk probability.

[0119] In some embodiments, according to the risk level, the warning information may include a warning level annotation and a handling suggestion, and may also be accompanied by a summary of the prediction data.

[0120] In some embodiments, the above-mentioned electrocardiogram data analysis device further includes an ACS warning push module, which is triggered when the prediction result exceeds a set threshold, and sends the warning information to relevant personnel through channels such as text messages, APP notifications, or emails.

[0121] In some embodiments, the above-mentioned electrocardiogram data analysis device further includes a smart terminal module, which is used to receive and convey the warning information to ensure that the monitored person and their relatives can be informed in a timely manner.

[0122] In some embodiments, the smart terminal module can notify the user in real time through visual and auditory means; it can also automatically contact emergency contacts or activate the automatic help function when necessary; it can also send all warning information and user response situations to the cloud server for storage by leveraging the advantages of cloud storage, for subsequent analysis and medical decision-making reference.

[0123] The division of each module in the above-mentioned electrocardiogram data analysis device based on multi-modal fusion is only for illustrative purposes. In other embodiments, the electrocardiogram data analysis device based on multi-modal fusion can be divided into different modules as needed to complete all or part of the functions of the above-mentioned electrocardiogram data analysis device based on multi-modal fusion.

[0124] For the specific limitations of the electrocardiogram data analysis device based on multi-modal fusion, reference can be made to the limitations of the electrocardiogram data analysis method based on multi-modal fusion in the above text, which will not be elaborated here. Each module in the above-mentioned electrocardiogram data analysis device based on multi-modal fusion can be implemented in whole or in part by software, hardware, and their combinations. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0125] Embodiment 3

[0126] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for electrocardiogram data analysis based on multi-modal fusion are implemented.

[0127] Embodiment 4

[0128] This embodiment provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the method for electrocardiogram data analysis based on multi-modal fusion are implemented.

[0129] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] Those skilled in the art can easily understand that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for analyzing electrocardiogram data based on multimodal fusion, characterized in that Including: Obtain multimodal data including time-series data and structural data; Preprocess the multimodal data; Extract local features and global features from the time-series data respectively, and fuse the extracted local features and global features to obtain a first feature vector; Fuse the features of the structural data to obtain a second feature vector; Fuse the first feature vector and the second feature vector to obtain a prediction vector; Determine the risk probability based on the prediction vector, and compare it with the risk threshold established by historical research. When the risk probability exceeds the threshold, generate a warning message.

2. The electrocardiogram data analysis method based on multi-modal fusion according to claim 1, wherein The obtaining of the multimodal data, which includes time-series data and structural data, includes: Collect time-series data including electrocardiogram signals and ballistocardiogram signals, and extract at least one piece of structural data including heart rate, respiratory rate, and blood oxygen saturation according to the ballistocardiogram signals; and, Collect at least one piece of structural data including age, gender, BMI index, presence or absence of a past history of cardiovascular and cerebrovascular diseases, presence or absence of smoking, and presence or absence of alcohol consumption.

3. The electrocardiogram data analysis method based on multi-modal fusion according to claim 1, wherein The preprocessing of the multimodal data includes: Divide the heartbeats of the time-series data, and perform standardization processing on the divided time-series data using the Z-score method; and, Perform standardization processing on continuous variables in the structural data using the Z-score method, and / or perform one-hot encoding on discrete variables in the structural data for standardization processing.

4. The electrocardiogram data analysis method based on multimodal fusion according to claim 1, wherein The extraction of local features and global features from the time-series data respectively includes: Input the time-series data into a multi-layer convolutional neural network model for processing to obtain local features; and, Input the time-series data into a multi-layer Transformer model for processing to obtain global features.

5. The electrocardiogram data analysis method based on multimodal fusion according to claim 4, wherein, The convolutional neural network model includes several convolutional blocks, where each convolutional block includes a convolutional layer, a normalization layer, an activation function layer, and a pooling layer; The convolutional layer is used to extract local features; The normalization layer is used to stabilize the training process of the model; The activation function layer is used to capture non-linear features; The pooling layer is used to reduce the feature dimension.

6. The electrocardiogram data analysis method based on multi-modal fusion according to claim 4, wherein, The Transformer model includes a position encoding layer, a multi-head attention layer, a feed-forward layer, a residual connection layer, and a normalization layer; The position encoding layer is used to encode the positions of the data; The multi-head attention layer is used to extract different features; The feed-forward layer is used to map the feature dimension; The residual connection layer is used to improve the convergence efficiency and training stability; The normalization layer is used to improve the training robustness and convergence speed.

7. The electrocardiogram data analysis method based on multimodal fusion according to claim 1, wherein Use a single-layer perceptron to extract high-dimensional features of time-series data to obtain a first feature vector; and use a multi-layer perceptron to extract high-dimensional features of structural data to obtain a second feature vector.

8. An electrocardiogram data analysis device based on multimodal fusion, characterized in that, Including: A data acquisition module for acquiring multimodal data, which includes time-series data and structural data; A data processing module for preprocessing the multimodal data; A feature extraction module, configured to respectively extract local features and global features from the time-series data, fuse the extracted local features and global features to obtain a first feature vector, and fuse the structural data to obtain a second feature vector; A feature fusion module, configured to fuse the first feature vector and the second feature vector to obtain a prediction vector; A risk prediction module, configured to determine a risk probability based on the prediction vector and compare it with a risk threshold established by historical research. When the risk probability exceeds the threshold, a warning message is generated.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images