Multi-mode electrocardiosignal and arrhythmia recognition system and recognition method

Abstract

The invention discloses a multi-modal electrocardiosignal and arrhythmia recognition system and method. The system comprises a signal acquisition module, a preprocessing module, a space-time alignment module, a feature extraction module, a self-adaptive fusion module, a multi-modal fusion module, a classification risk assessment module and a visual report module. The recognition method comprises the following steps: S1, synchronously acquiring an electrocardiosignal and a heart sound signal, and preprocessing the electrocardiosignal and the heart sound signal; s2, establishing a time corresponding relation between the electrical activity and the mechanical activity; s3, calculating a mechanical shrinkage efficiency index; s4, evaluating the quality of the electrocardiosignal and the heart sound signal in real time, and dynamically adjusting contribution weights of the electrocardiosignal and the heart sound signal in a fusion process according to the quality; s5, adopting a multi-head attention mechanism to generate a joint feature vector for classification; s6, inputting the joint feature vector into a classification network model, and outputting an arrhythmia type and a quantitative risk level at the same time by the classification network model; and S7, generating a visual atlas and a structured diagnosis report.

Description

Technical Field

[0001] This invention relates to the field of electrocardiogram (ECG) signal analysis, and more particularly to a multimodal ECG signal and arrhythmia recognition system and method. Background Technology

[0002] With the rising incidence of cardiovascular disease, early diagnosis and risk assessment of arrhythmias are becoming increasingly important. Current Holter monitoring systems primarily rely on the analysis of ECG waveforms and rhythms to diagnose arrhythmias. However, the heart is an organ with tightly coupled electrophysiological and mechanical activity; under certain pathological conditions, temporary or compensatory decoupling between its electrical activity and mechanical function can occur.

[0003] This leads to a clinical challenge: traditional, standalone electrocardiogram (ECG) analysis techniques are insufficient to effectively detect occult arrhythmic events that present with atypical electrical signals but have already triggered compensatory abnormalities in cardiac mechanical function. Although these events do not form a persistent, typical abnormal pattern on the ECG, they may be early signs of deteriorating cardiac function, resulting in missed diagnoses and delayed risk assessments, failing to meet the clinical need for early warning of high-risk patients.

[0004] Therefore, there is an urgent need for a new diagnostic system and method that can integrate multimodal physiological signals, especially synchronously analyze electrocardiogram and heart sound (PCG) signals and quantify their coupling relationship, in order to achieve early identification and accurate risk assessment of occult arrhythmias. Summary of the Invention

[0005] This invention aims to address the problem that existing technologies rely on single electrocardiogram analysis or simple multimodal fusion, which are difficult to effectively identify occult arrhythmias caused by electromechanical decoupling, thus leading to missed diagnoses and delayed risk warnings. It provides a multimodal electrocardiogram signal and arrhythmia recognition system and method to improve detection accuracy and avoid missed diagnoses.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A multimodal electrocardiogram (ECG) signal and arrhythmia recognition system includes a signal acquisition module for signal acquisition, a preprocessing module for preprocessing the acquired signals, a spatiotemporal alignment module for spatiotemporal alignment, a feature extraction module for signal feature extraction, an adaptive fusion module for signal fusion weight adjustment, a multimodal fusion module for signal fusion, a classification risk assessment module for arrhythmia classification and risk assessment, and a visualization report module for generating a visualization report. The signal output of the signal acquisition module is connected to the signal input of the preprocessing module, the preprocessing module is connected to the signal input of the spatiotemporal alignment module, the spatiotemporal alignment module is connected to the signal input of the feature extraction module, the feature extraction module is connected to the signal input of the adaptive fusion module, the adaptive fusion module is connected to the signal input of the multimodal fusion module, the multimodal fusion module is connected to the signal input of the classification risk assessment module, and the classification risk assessment module is connected to the signal input of the visualization report module.

[0008] A method for identifying multimodal electrocardiogram signals and arrhythmias includes the following steps:

[0009] S1. The device synchronously acquires 12-lead electrocardiogram (ECG) signals and heart sound signals, and performs filtering and noise reduction processing on the ECG and heart sound signals respectively;

[0010] S2. Use the ECG R wave to locate the heartbeat cycle, and based on this, segment and align the first and second heart sounds in the heart sound signal to establish the time correspondence between electrical activity and mechanical activity.

[0011] S3. Extract the morphological rhythm features of the electrocardiogram signal and the time-frequency features of the heart sound signal, and calculate the mechanical contractile efficiency index;

[0012] S4. Real-time assessment of the quality of ECG and heart sound signals, and dynamic adjustment of their contribution weights in the fusion process accordingly;

[0013] S5. Employ a multi-head attention mechanism to allow the ECG feature vector and heart sound feature vector to interact, automatically weighted and filtered out the fusion information related to diagnosis, and generate a joint feature vector for classification.

[0014] S6. Input the joint feature vector into the classification network model. The classification network model simultaneously outputs the specific arrhythmia type and the quantitative risk level based on the coupled features, realizing the integration of diagnosis and prognostic assessment.

[0015] S7 generates intuitive graphs containing signal-aligned waveforms and decoupling markers, as well as structured diagnostic reports, to provide decision support for physicians.

[0016] Furthermore, step S1 specifically includes the following steps:

[0017] S11. Synchronously acquire 12-lead electrocardiogram signals and heart sound signals through integrated sensing devices;

[0018] S12. Perform baseline drift removal and power frequency interference filtering on the electrocardiogram signal;

[0019] S13. Perform segmented filtering on the heart sound signal to retain the main energy frequency band; and apply wavelet threshold denoising to the heart sound signal to suppress environmental noise.

[0020] Furthermore, step S2 specifically includes the following steps:

[0021] S21. Detect the peak value of the R wave in the electrocardiogram signal to locate the starting point of each cardiac cycle;

[0022] S22. Locate the first and second heart sounds in the heart sound signal. Based on the physiological delay between the R wave of the electrocardiogram signal and the first heart sound, search for the envelope extreme point of the heart sound signal within the time window after the R wave of the electrocardiogram signal, and align the heart rhythm of the electrocardiogram signal and the heart sound signal through dynamic time warping.

[0023] S23. Divide each cardiac cycle into systolic and diastolic phases, and label the corresponding multimodal data segments.

[0024] Furthermore, step S3 specifically includes the following steps:

[0025] S31. Extract morphological rhythm features from the preprocessed electrocardiogram (ECG) signal. The morphological rhythm features include ECG feature vectors. P wave duration QRS group width ST segment slope and the standard deviation of the interval between R waves The ratio of the intervals between adjacent R waves ;

[0026] S32. Perform feature analysis on the preprocessed heart sound signal to extract the heart sound feature vector. Calculate the energy ratio of the first heart sound to the second heart sound. During the contraction interval The energy concentration of the first central frequency band is obtained through spectral analysis, and its calculation formula is as follows:

[0027]

[0028] In the formula, The first core frequency band has high energy concentration. Let be the power spectral density function of the heart sound signal;

[0029] S33. Based on the time difference between the peak value of the R wave in the electrocardiogram signal and the peak value of the first heart sound in the heart sound signal. The mechanical shrinkage efficiency index is constructed, and its calculation formula is as follows:

[0030]

[0031] In the formula, The mechanical shrinkage efficiency index. The QRS group width, The energy of the first heartbeat, This is the energy of the second heartbeat.

[0032] Furthermore, step S4 specifically includes the following steps:

[0033] S41. Perform real-time quality assessment calculations on the preprocessed ECG and heart sound signals to obtain quality scores for the ECG and heart sound signals.

[0034] S42. Normalize the quality scores of the electrocardiogram (ECG) and heart sound signals to the range [0,1]; the calculation formula is as follows:

[0035]

[0036] In the formula, For the fusion weights assigned to the ECG feature vectors, The fusion weights assigned to the heart sound feature vector;

[0037] S43, Regarding ECG feature vectors Heart sound feature vector Weighted processing is performed to generate a weighted feature vector of electrocardiogram. Weighted feature vector of heart sounds The calculation formula is as follows:

[0038] .

[0039] Furthermore, step S41 specifically includes the following steps:

[0040] S411. Calculate the effective amplitude ratio of the electrocardiogram signal. The calculation formula is as follows:

[0041]

[0042] In the formula, The amplitude difference is the difference between the start of the P wave and the end of the T wave in the electrocardiogram (ECG) signal. The global amplitude range is the amplitude difference of the entire ECG signal.

[0043] S412. The variance of the R-wave position in the electrocardiogram signal is calculated based on the Pan-Tompkins algorithm as the confidence level of R-wave detection.

[0044] S413, Effective amplitude ratio based on electrocardiogram signal The quality score of the electrocardiogram signal is calculated based on the R-wave detection confidence level; the calculation formula is as follows:

[0045] ;

[0046] S414. Calculate the energy ratio stability (ESR) of the first heart sound / first heart sound ratio; the formula is as follows:

[0047]

[0048] In the formula, Calculated based on several heartbeat cycles The average value obtained by calculating the ratio For the same series of cardiac cycles The standard deviation calculated from the ratio;

[0049] S415. Calculate the quality fraction of the heart sound signal; the formula is as follows:

[0050] ;

[0051] In the formula, This represents the signal-to-noise ratio of the heart sound signal.

[0052] Furthermore, step S5 specifically includes the following steps:

[0053] S51. Weighted feature vector of electrocardiogram Weighted feature vector of heart sounds As input, a query matrix is ​​generated through linear transformation. Key matrix Sum matrix ;

[0054] S52, query matrix Key matrix Sum matrix The calculation is based on a multi-head attention mechanism, and the output obtains an ECG feature representation weighted by heart sound signals; the calculation formula is as follows:

[0055]

[0056] In the formula, This represents the weighted electrocardiogram (ECG) characteristics of the heart sound signals. Key matrix The transpose of the matrix, Key matrix The dimension;

[0057] S53. Compare the weighted ECG feature representation of the heart sound signal with the weighted ECG feature vector. The residuals are summed, and the summation result is then normalized to generate a joint feature vector that integrates multimodal information. The calculation formula is as follows:

[0058]

[0059] In the formula, The fused joint feature vector This is a layer normalization operation.

[0060] Furthermore, step S6 specifically includes the following steps:

[0061] S61. Combine the fused joint feature vectors The data is fed into a classification network model with a two-dimensional convolutional neural network structure for in-depth analysis. The classification network model generates logical values ​​for classification through the extraction of convolutional kernels and the pooling operation of the global pooling layer.

[0062] S62. The output layer of the classification network model uses the Sigmoid activation function to classify the specific types of arrhythmias; at the same time, it assesses and quantifies the risk level based on the energy concentration of the first cardiac band, the mechanical contraction efficiency index, and a preset clinical experience threshold.

[0063] Furthermore, step S7 includes the following steps:

[0064] S71. Generate and display the spatiotemporal alignment results of the R wave in the electrocardiogram signal and the first / second heart sound in the heart sound signal based on the output results, and mark the decoupling region;

[0065] S72. Output a structured report, which includes the type of arrhythmia, risk level, and timestamps of abnormal coupling events.

[0066] Compared with the prior art, the advantages and positive effects of this invention are:

[0067] 1. This invention, through deep coupling analysis of electrocardiogram (ECG) signals and heart sound signals, can reveal abnormal correlations between cardiac electrophysiological and mechanical activities. This mechanism can effectively identify occult arrhythmia events that are weak or transient in electrical signals but have already triggered abnormal mechanical compensation. By extracting and quantifying characteristic indicators that characterize the efficiency of electromechanical coupling, it overcomes the deficiency of insufficient information in a single modality, thereby extracting more pathologically significant features from complex physiological signals, reducing the probability of missed diagnosis due to signal noise, baseline drift, or transient events, and providing a more reliable technical means for early diagnosis.

[0068] 2. By continuously monitoring the dynamic changes in the electromechanical coupling state, this invention can detect early signs of the decline in cardiac compensatory function, even if typical and persistent arrhythmia waveforms have not yet appeared. This ability to intervene at an earlier stage of the physiological and pathological process allows for earlier clinical intervention, providing key time and information advantages for developing preventive treatment strategies and adjusting medication regimens, and realizing a shift from passive diagnosis to proactive risk assessment.

[0069] 3. This invention employs a mechanism of mutual verification and supplementation of multimodal signals, which improves the reliability and confidence of data and results in long-term monitoring in uncontrolled environments such as homes. When the quality of a certain modality signal temporarily deteriorates due to motion artifacts or noise interference, it can be used to make auxiliary judgments and corrections based on the signal of another modality. For example, the temporal information of the heart sound signal can be used to verify or complete the ambiguous R-wave location in the electrocardiogram signal. This inherent cross-validation capability reduces the over-reliance on the quality of a single signal, lowers the incidence of false alarms and invalid reports, ensures the continuity and availability of long-term monitoring data, and better meets the needs of practical application scenarios.

[0070] 4. This invention improves the reliability and robustness of diagnostic results in non-ideal monitoring environments by introducing an adaptive weight allocation mechanism based on dynamic signal quality assessment. This mechanism can sense changes in the quality of ECG and heart sound signals in real time and automatically adjust their contribution to decision fusion, thereby reducing the risk of misjudgment or missed judgment caused by temporary degradation of a single modality signal. In addition, by providing real-time signal quality feedback and fusion confidence indicators, it not only enhances the clinical interpretability of the system output results, but also provides clear guidance for users to actively optimize measurement conditions, ultimately achieving continuous and stable high-performance monitoring results in real-world scenarios. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a system framework diagram of the present invention;

[0073] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art to all other embodiments obtained without creative effort should be included within the protection scope of the present invention.

[0075] This invention provides a multimodal ECG signal and arrhythmia identification system and method capable of synchronously acquiring, precisely aligning, and deeply fusing ECG and heart sound signals, and quantifying their coupling state, so as to achieve early and accurate diagnosis and risk assessment of arrhythmias.

[0076] like Figure 1 As shown, the multimodal ECG signal and arrhythmia recognition system includes a signal acquisition module, a preprocessing module, a spatiotemporal alignment module, a feature extraction module, a multimodal fusion module, a classification and risk assessment module, and a visualization and reporting module; the signal acquisition module, preprocessing module, spatiotemporal alignment module, feature extraction module, multimodal fusion module, classification and risk assessment module, and visualization and reporting module are connected in sequence.

[0077] The functions of each module are as follows:

[0078] Signal acquisition module: responsible for synchronously acquiring 12-lead ECG signals and heart sound signals (PCG), ensuring time synchronization error ≤1ms;

[0079] Preprocessing module: performs baseline drift removal and power frequency interference filtering on ECG signals; performs wavelet threshold denoising and frequency band filtering on PCG signals;

[0080] Spatiotemporal alignment module: detects the R-wave peak of the ECG signal and the S1 / S2 peak of the PCG signal, and realizes spatiotemporal alignment of multimodal signals through dynamic time warping (DTW);

[0081] Feature extraction module: Extracts morphological and rhythmic features from ECG signals, extracts time-domain and frequency-domain features from PCG signals, and calculates coupling features (such as electromechanical delay and mechanical contraction efficiency index).

[0082] Adaptive fusion module: Real-time evaluation of ECG and PCG signal quality, and dynamic adjustment of fusion weights;

[0083] Multimodal fusion module: Uses a multi-head attention mechanism to fuse ECG and PCG features to generate a joint feature vector;

[0084] Classification and Risk Assessment Module: Based on fused features, a lightweight convolutional network is used for arrhythmia classification and risk assessment;

[0085] Visualization Report Module: Generates synchronized overlay waveforms and structured reports, providing diagnostic feedback.

[0086] like Figure 2 As shown, the multimodal electrocardiogram signal and arrhythmia identification method includes the following steps:

[0087] Step 1, Signal Acquisition and Preprocessing, is performed by the signal acquisition module and the preprocessing module: 12-lead electrocardiogram (ECG) and heart sound (PCG) signals are acquired simultaneously through the device, and filtered and denoised respectively to provide a data foundation for subsequent analysis;

[0088] Step 1, the signal acquisition and preprocessing operation specifically includes the following steps:

[0089] Step 11: Synchronously acquire 12-lead ECG signals and PCG heart sound signals using integrated sensing devices. The device has a built-in high-precision clock to ensure that the signal time synchronization error is ≤1ms.

[0090] Step 12: Perform baseline drift removal (using a bidirectional high-pass filter with a cutoff frequency of 0.5Hz) and power frequency interference filtering (50Hz notch filter) on the ECG signal.

[0091] Step 13: Perform segmented filtering on the PCG signal: retain the main energy frequency band of heart sounds (20Hz–150Hz), and use wavelet threshold denoising (Daubechies 6 wavelet, Level 5 decomposition) to suppress environmental noise;

[0092] This step acquires high-quality, highly synchronized raw electrocardiogram (ECG) and heart sound (PCG) signals, laying a reliable data foundation for subsequent analysis. Dedicated integrated sensing devices ensure that the two signals are strictly aligned in time, and targeted filtering algorithms are used to remove baseline drift, power line interference, and environmental noise from the PCG, thereby purifying the signals and retaining key information related to cardiac activity.

[0093] Step 2, Spatiotemporal alignment operation, performed by the spatiotemporal alignment module: using the ECG R wave to locate the heartbeat cycle, and accordingly segmenting and aligning the first and second heart sounds in the heart sound signal to establish the temporal correspondence between electrical activity and mechanical activity;

[0094] Step 2, the spatiotemporal alignment operation specifically includes the following steps:

[0095] Step 21: Detect the R-wave peak value of the ECG signal (using the Pan-Tompkins algorithm) to locate the starting point of each heartbeat;

[0096] Step 22: Locate the first heart sound (S1) and the second heart sound (S2) in the PCG signal: Based on the physiological delay between the ECGR wave and S1 (usually 50ms–100ms), search for the extreme point of the PCG envelope within the time window after the R wave, and align the ECG and PCG cycles through dynamic time warping (DTW).

[0097] Step 23: Define spatiotemporal synchronization labels: Divide each heartbeat cycle into systolic (S1 to S2) and diastolic (S2 to the next S1), and label the corresponding multimodal data segments;

[0098] This step establishes a precise temporal correspondence for subsequent joint analysis; by detecting the R wave in ECG and the first heart sound (S1) and second heart sound (S2) in PCG, each heartbeat cycle is divided into various phases of electrical activity (ECG) and mechanical activity (PCG) (such as systole and diastole), achieving precise matching and synchronization of the two heterogeneous signals within the same physiological event framework.

[0099] Step 3: Joint feature extraction operation, performed by the feature extraction module: extract the morphological rhythm features of the electrocardiogram signal and the time-frequency features of the heart sound signal respectively, and calculate the electromechanical delay and mechanical contraction efficiency index to quantify the electromechanical coupling state of the heart;

[0100] Step 3, the joint feature extraction operation specifically includes the following steps:

[0101] Step 31: Extract key morphological rhythm features from the preprocessed electrocardiogram (ECG) signal, specifically including ECG feature vectors. P wave duration QRS group width ST segment slope Morphological parameters, including RR interval standard deviation Ratio of adjacent RR intervals Rhythmic indicators, including those included;

[0102] Step 32: Perform feature analysis on the synchronously acquired heart sound signals (PCG) and extract heart sound feature vectors. Calculate the energy ratio of S1 to S2. During the contraction interval Isochronous time domain characteristics, and the energy concentration of the S1 band was obtained through spectrum analysis:

[0103]

[0104] In the formula, The energy concentration in the S1 frequency band. Let be the power spectral density function of the heart sound signal. For frequency;

[0105] Step 33: Calculate the electromechanical coupling index based on the above characteristics, including the time difference from the peak value of the ECG R wave to the peak value of the PCG S1 wave. (Electromechanical delay), and further integrate morphological and energy information to construct a mechanical contraction efficiency index:

[0106]

[0107] In the formula, The mechanical shrinkage efficiency index. For QRS group time limit, For electromechanical delay, The energy of the first heart sound (S1), The energy for the second heart sound (S2);

[0108] This index comprehensively quantifies the efficiency of the heart's electrical activity in converting into mechanical contraction, providing key evidence for core diagnosis;

[0109] This step extracts quantitative features from the aligned signals that reveal the electromechanical coupling relationship. The morphological rhythm features of ECG and the time-frequency domain features of PCG are extracted respectively, and coupling features such as "electromechanical delay" and "mechanical contraction efficiency index (η)" are calculated. These features directly reflect the efficiency and synchronicity of the conversion of cardiac electrical excitation into mechanical contraction, which is the core of identifying hidden functional disorders.

[0110] Step 4: Adaptive fusion optimization operation, performed by the adaptive fusion module: The quality of ECG and heart sound signals is evaluated in real time before signal fusion, and the contribution weights of the two in the fusion process are dynamically adjusted accordingly.

[0111] Step 4, the adaptive fusion optimization operation specifically includes the following steps:

[0112] Step 41: Perform real-time quality assessment on the preprocessed ECG and PCG signals, specifically as follows:

[0113] Step 411: Calculate the effective amplitude ratio of the ECG segment. :

[0114]

[0115] In the formula, The amplitude difference is the difference between the start of the P wave and the end of the T wave; the global amplitude range is the amplitude difference of the entire signal segment.

[0116] Calculate the R-wave detection confidence (RRC): Based on the Pan-Tompkins algorithm, output the R-wave position variance. The smaller the variance, the higher the confidence.

[0117] Based on the obtained effective amplitude ratio and R-wave detection confidence, the ECG quality score is calculated: ;

[0118] Step 412: Based on the S1 / S2 energy ratio and signal-to-noise ratio evaluation, calculate the S1 / S2 energy ratio stability ESR:

[0119]

[0120] In the formula, To calculate from several recent cardiac cycles The average value obtained by calculating the ratio It is the energy of the first heart sound in a heartbeat cycle (usually estimated by summing the squares of the signal during the time period of S1 or by calculating its root mean square). The energy of the second heart sound within the same cardiac cycle. For recent series of cardiac cycles The standard deviation calculated from the ratio;

[0121] PCG quality score ( (Signal-to-noise ratio of heart sound signal)

[0122] Step 413: When the quality score of a signal is lower than the threshold (e.g., Q<0.3), the signal is temporarily isolated, and only another modal signal is used for single-modal prediction. At the same time, a user prompt is triggered (e.g., "Please adjust the position of the heart sound sensor").

[0123] Step 42: Normalize the mass fraction to the range [0,1]:

[0124]

[0125] In the formula, For the fusion weights assigned to ECG features, The fusion weights assigned to the heart sound features;

[0126] Use weights on the ECG feature vector output from step 3 Heart sound feature vector Preprocessing is performed to generate weighted feature vectors. and : .

[0127] Step 5: Multimodal fusion operation, performed by the multimodal fusion module: Employing a multi-head attention mechanism, ECG and heart sound features interact, automatically weighting and filtering out the fusion information most relevant to diagnosis, and generating a joint feature vector for classification;

[0128] Step 5, the multimodal fusion operation specifically includes the following steps:

[0129] Step 51: Convert the preprocessed ECG feature vector Heart sound feature vector As input, a query matrix is ​​generated through linear transformation. Key matrix Sum matrix , All are learnable weight matrices;

[0130] Step 52: Calculate the matrix using a multi-head attention mechanism to obtain the ECG feature representation weighted based on the heart sound context:

[0131]

[0132] In the formula, This is the feature representation after weighted fusion. For querying the matrix, The key matrix, Key matrix The transpose of the matrix, For value matrices, Key vector The dimension;

[0133] Step 53: Compare the attention output with the original ECG characteristics The residuals are summed, and the summation result is then normalized layer by layer to generate a joint feature vector that integrates multimodal information, providing highly discriminative features for subsequent classification.

[0134]

[0135] In the formula, The fused joint feature vector For layer normalization operation;

[0136] This step intelligently integrates feature information from ECG and PCG, rather than simply splicing them together; it uses an attention mechanism to dynamically weigh the importance and correlation of the two signal features, automatically focusing on the modal features most relevant to the current abnormality, thereby generating a multimodal fusion feature vector that can more comprehensively and accurately represent the cardiac state.

[0137] Step 6, Classification and Risk Assessment, is performed by the classification and risk assessment module: the fused features are input into the classification network, and the specific arrhythmia type and the quantitative risk level based on the coupled features are output, realizing the integration of diagnosis and prognostic assessment.

[0138] Step 6, the classification and risk assessment operation, specifically includes the following steps:

[0139] Step 61: Generate the fused feature vector The input is fed into a lightweight 2D convolutional neural network (2-layer 1D-CNN) for deep pattern mining. The network extracts local temporal patterns hidden in the fused features through convolutional kernels, and aggregates local features into global, fixed-length feature representations through global pooling layers to generate logical values ​​for classification.

[0140] Step 62: The output layer uses the Sigmoid activation function to perform the following two tasks simultaneously:

[0141] Classify specific types of arrhythmias (such as atrial fibrillation, atrial tachycardia, and conduction block);

[0142] Risk assessment is conducted (classified into low, medium, and high levels). The determination of the risk level not only depends on the network output but also refers to the coupling characteristics and the value of the mechanical contraction efficiency index. When the value is lower than the clinical experience threshold, a high risk level will be directly indicated, thereby achieving integrated output of diagnosis and prognosis assessment.

[0143] This step can make a final diagnostic decision based on the fused high-level features; identify specific types of arrhythmias (such as atrial fibrillation and atrial tachycardia), and combine coupled features (such as the η index) to classify patients by risk, thus achieving a comprehensive diagnostic upgrade from traditional event detection to one that combines event identification and functional assessment.

[0144] Step 7, Visual Feedback Operation, operated by the Visual Reporting Module: Generates intuitive graphs containing signal alignment waveforms and decoupling markers, as well as structured diagnostic reports, to provide decision support for doctors.

[0145] Step 7, the visual feedback operation specifically includes the following steps:

[0146] Step 71: Generate a synchronized overlay waveform diagram, displaying the spatiotemporal alignment results of the ECG R wave and PCG S1 / S2, and marking the decoupling regions (e.g., );

[0147] Step 72: Output a structured report, including the type of arrhythmia, risk level, timestamps of abnormal coupling events, and confidence levels;

[0148] This step transforms the system's analysis results into intuitive and actionable clinical information; by generating waveforms that display signal alignment and anomaly annotations, as well as structured diagnostic reports, it provides doctors with clear diagnostic evidence and risk assessment results, bridging the gap between algorithm analysis and clinical application, and assisting doctors in making final decisions.

[0149] This invention utilizes innovative synchronization alignment technology and coupled feature quantization formulas ( Based on attention-based fusion models, a system and method were constructed to achieve electromechanical coupling analysis and output integrated diagnostic risk results. Based on the detection of R waves in ECG and S1 / S2 peaks in PCG, combined with dynamic time warping (DTW) technology, it can achieve precise periodic division and alignment of ECG and heart sound signals. Furthermore, it evaluates the quality of ECG and heart sound signals in real time, uses the attention mechanism to dynamically calculate the correlation weight between ECG and PCG features, realizes context-aware feature fusion, and generates joint feature vectors instead of simple splicing, thereby improving the accuracy of diagnostic results.

[0150] The present invention has the following beneficial effects:

[0151] 1. This invention, through deep coupling analysis of electrocardiogram (ECG) signals and heart sound signals, can reveal abnormal correlations between cardiac electrophysiological and mechanical activities. This mechanism can effectively identify occult arrhythmia events that are weak or transient in electrical signals but have already triggered abnormal mechanical compensation. By extracting and quantifying characteristic indicators that characterize the efficiency of electromechanical coupling, it overcomes the deficiency of insufficient information in a single modality, thereby extracting more pathologically significant features from complex physiological signals, reducing the probability of missed diagnosis due to signal noise, baseline drift, or transient events, and providing a more reliable technical means for early diagnosis.

[0152] 2. By continuously monitoring the dynamic changes in the electromechanical coupling state, this invention can detect early signs of the decline in cardiac compensatory function, even if typical and persistent arrhythmia waveforms have not yet appeared. This ability to intervene at an earlier stage of the physiological and pathological process allows for earlier clinical intervention, providing key time and information advantages for developing preventive treatment strategies and adjusting medication regimens, and realizing a shift from passive diagnosis to proactive risk assessment.

[0153] 3. This invention employs a mechanism of mutual verification and supplementation of multimodal signals, which improves the reliability and confidence of data and results in long-term monitoring in uncontrolled environments such as homes. When the quality of a certain modality signal temporarily deteriorates due to motion artifacts or noise interference, it can be used to make auxiliary judgments and corrections based on the signal of another modality. For example, the temporal information of the heart sound signal can be used to verify or complete the ambiguous R-wave location in the electrocardiogram signal. This inherent cross-validation capability reduces the over-reliance on the quality of a single signal, lowers the incidence of false alarms and invalid reports, ensures the continuity and availability of long-term monitoring data, and better meets the needs of practical application scenarios.

[0154] 4. This invention improves the reliability and robustness of diagnostic results in non-ideal monitoring environments by introducing an adaptive weight allocation mechanism based on dynamic signal quality assessment. This mechanism can sense changes in the quality of ECG and heart sound signals in real time and automatically adjust their contribution to decision fusion, thereby reducing the risk of misjudgment or missed judgment caused by temporary degradation of a single modality signal. In addition, by providing real-time signal quality feedback and fusion confidence indicators, it not only enhances the clinical interpretability of the system output results, but also provides clear guidance for users to actively optimize measurement conditions, ultimately achieving continuous and stable high-performance monitoring results in real-world scenarios.

Claims

1. A multi-modal electrocardiosignal and arrhythmia recognition system, characterized by: The system comprises a signal acquisition module for signal acquisition, a preprocessing module for preprocessing operation on the acquired signal, a space-time alignment module for space-time alignment operation, a feature extraction module for signal feature extraction, an adaptive fusion module for signal fusion weight adjustment, a multi-modal fusion module for signal fusion, a classification and risk assessment module for arrhythmia classification and risk assessment, and a visualization report module for generating a visualization report. The signal output end of the signal acquisition module is connected with the signal input end of the preprocessing module, the signal output end of the preprocessing module is connected with the signal input end of the space-time alignment module, the signal output end of the space-time alignment module is connected with the signal input end of the feature extraction module, the signal output end of the feature extraction module is connected with the signal input end of the adaptive fusion module, the signal output end of the adaptive fusion module is connected with the signal input end of the multi-modal fusion module, the signal output end of the multi-modal fusion module is connected with the signal input end of the classification and risk assessment module, and the signal output end of the classification and risk assessment module is connected with the signal input end of the visualization report module.

2. The method of claim 1, wherein the method is performed by the system of claim 1. The method comprises the following steps: S1, 12-lead ECG signals and heart sound signals are synchronously acquired by a device, and the ECG signals and the heart sound signals are respectively subjected to filtering and denoising processing; S2, the ECG R wave is positioned to locate the start point of each cardiac cycle, and the first heart sound and the second heart sound in the heart sound signal are segmented and aligned according to the ECG R wave to establish the time correspondence between electrical activity and mechanical activity; S3, the morphological rhythm features of the ECG signals and the time-frequency features of the heart sound signals are extracted, and a mechanical systolic efficiency index is calculated; S4, the quality of the ECG signals and the heart sound signals is evaluated in real time, and the contribution weight of the ECG signals and the heart sound signals in the fusion process is dynamically adjusted according to the quality; S5, a multi-head attention mechanism is adopted to enable the ECG feature vector and the heart sound feature vector to interact, and the fusion information related to diagnosis is automatically weighted and selected to generate a joint feature vector for classification; S6, the joint feature vector is input into a classification network model, the classification network model simultaneously outputs a specific arrhythmia type and a quantitative risk level based on coupling features, and diagnosis and prognosis evaluation are integrated; S7, an intuitive atlas containing signal alignment waveform and decoupling mark and a structured diagnosis report are generated to provide decision support for doctors.

3. The multi-modal cardiac electrical signal and arrhythmia identification method of claim 2, wherein: The step S1 specifically comprises the following steps: S11, 12-lead ECG signals and heart sound signals are synchronously acquired by an integrated sensing device; S12, baseline drift removal operation and power frequency interference filtering operation are performed on the ECG signals; S13, segment filtering operation is performed on the heart sound signals to retain the main energy frequency band, and wavelet threshold denoising operation is performed on the heart sound signals to suppress environmental noise.

4. The multi-modal cardiac electrical signal and arrhythmia identification method of claim 2, wherein: The step S2 specifically comprises the following steps: S21, the R wave peak of the ECG signal is detected to locate the start point of each cardiac cycle; S22, locate the first heart sound and the second heart sound in the heart sound signal, search for the envelope extreme point of the heart sound signal in a time window after the R wave of the electrocardiogram signal based on the physiological delay of the R wave of the electrocardiogram signal and the first heart sound, and align the heart beat cycle of the electrocardiogram signal and the heart sound signal through dynamic time warping; S23, divide each heart beat cycle into a systole period and a diastole period, and mark the multi-modal data segment of the corresponding period.

5. The multi-modal cardiac electrical signal and arrhythmia identification method of claim 2, wherein: The step S3 specifically comprises the following steps: S31, extract morphological rhythm features from the preprocessed electrocardio signal, the morphological rhythm features include electrocardio feature vectors , P-wave duration , QRS complex width , ST segment slope , and standard deviation of the interval between R-waves , interval ratio between adjacent R-waves ; S32, carry out feature analysis on the preprocessed heart sound signal, and extract a heart sound feature vector , calculate the energy ratio of the first heart sound and the second heart sound , systolic interval , and obtain the first heart sound frequency band energy concentration degree through spectrum analysis, and the calculation formula is: wherein is a first heart sound frequency band energy concentration, is a power spectral density function of the heart sound signal; S33、According to the time difference from the R-wave peak value of the electrocardiosignal to the first heart sound peak value of the heart sound signal , a mechanical systolic efficiency index is constructed, and the calculation formula is: wherein is a mechanical contraction efficiency index, is a QRS complex width, is an energy of the first heart sound, is an energy of the second heart sound.

6. The multi-modal cardiac electrical signal and arrhythmia identification method of claim 2, wherein: The step S4 specifically comprises the following steps: S41, perform real-time quality evaluation calculation on the preprocessed electrocardiogram signal and heart sound signal to obtain the quality score of the electrocardiogram signal and the heart sound signal; S42, normalize the quality score of the electrocardiogram signal and the heart sound signal to the range of [0, 1]; the calculation formula is: In the formula, is a fusion weight assigned to the electrocardio feature vector, is a fusion weight assigned to the heart sound feature vector; S43, the electrocardio feature vector and the heart sound feature vector weighted processing, generating the electrocardio weighted feature vector and the heart sound weighted feature vector The calculation formula is: 。 7. The multi-modal cardiac electrical signal and arrhythmia identification method of claim 6, wherein: The step S41 specifically comprises the following steps: S411、Calculate the effective amplitude ratio of the electrocardiosignal ; the calculation formula is: In the formula, is the amplitude difference in the period from the beginning of the P wave to the end of the T wave in the electrocardiosignal, and the global amplitude range is the amplitude difference of the entire electrocardiosignal. S412, calculate the R wave position variance in the electrocardiogram signal as the R wave detection confidence based on the Pan-Tompkins algorithm; S413、based on the effective amplitude ratio of the electrocardiosignal and the R-wave detection confidence, a quality score of the electrocardiosignal is calculated; the calculation formula is: ； S414, calculate the energy ratio stability ESR of the first heart sound / first heart sound; the calculation formula is: wherein is the average of the ratio calculated over several heart cycles is the average of the ratio calculated over several heart cycles is the standard deviation of the ratio calculated over the same series of heart cycles is the standard deviation of the ratio calculated over the same series of heart cycles S415, calculate the quality score of the heart sound signal; the calculation formula is: ； In the formula, is the signal-to-noise ratio of the heart sound signal.

8. The multi-modal cardiac electrical signal and arrhythmia identification method of claim 2, wherein: The step S5 specifically comprises the following steps: S51, the electrocardio weighted feature vector and the heart sound weighted feature vector As input, generate a query matrix through linear transformation , a key matrix and a value matrix ; S52, the query matrix , the key matrix , and the value matrix Based on the multi-head attention mechanism, the output obtains the heart sound signal weighted electrocardio feature representation; the calculation formula is: wherein is a weighted electrocardiographic feature representation of the heart sound signal, is a key matrix is a transpose matrix of the key matrix is a key matrix is a dimension of the key matrix S53. Compare the weighted ECG feature representation of the heart sound signal with the weighted ECG feature vector. The residuals are summed, and the summation result is then normalized to generate a joint feature vector that integrates multimodal information. The calculation formula is as follows: In the formula, is the fused joint feature vector, is a layer normalization operation.

9. The multi-modal cardiac electrical signal and arrhythmia identification method of claim 2, wherein: The step S6 specifically comprises the following steps: S61、the fused joint feature vector The input is input into a classification network model of a two-dimensional convolutional neural network structure for deep mining, and the classification network model generates a logical value for classification through extraction operation of a convolution kernel and pooling operation of a global pooling layer. S62, the output layer of the classification network model adopts a Sigmoid activation function to classify the specific type of arrhythmia; at the same time, the first heart sound frequency band energy concentration degree, the mechanical systolic efficiency index, and the preset clinical experience threshold evaluation are used to quantize the risk grade.

10. The multi-modal cardiac electrical signal and arrhythmia identification method of claim 2, wherein: The step S7 comprises the following steps: S71, generate a display of the spatiotemporal alignment result of the R wave in the electrocardiogram signal and the first heart sound / second heart sound in the heart sound signal according to the output result, and mark the decoupling region; S72, output a structured report, and the structured report contains the arrhythmia type, the risk grade, and the timestamp of the abnormal coupling event.

Images