Multimodal flexible wearable cardiac function monitoring methods, systems, and devices

Abstract

This invention discloses a flexible wearable method, system, and device for monitoring cardiac function based on multimodal sensing, relating to the field of health monitoring technology. The method includes: when a flexible patch sensing module is placed in the detection area, multimodal signals are output through the flexible patch sensing module; wherein the flexible patch sensing module includes an electrocardiogram sensor, a pulse wave sensor, a heart sound sensor, a strain sensor, and a microfluidic biosensor. This invention achieves high-precision synchronous signal acquisition by integrating multimodal sensing through a flexible patch, adaptively extracts multi-source cardiac signal features using a multi-scale convolutional neural model, and captures temporal correlations using a bidirectional long short-term memory network, significantly improving the accuracy and reliability of cardiac disease diagnosis, while supporting non-invasive, continuous monitoring and early pathological screening.

Description

Technical Field

[0001] This invention relates to the field of health monitoring technology, specifically to a flexible wearable method, system, and device for monitoring cardiac function based on multimodality. Background Technology

[0002] In the clinical diagnosis of cardiovascular diseases, cardiac electrophysiological signals (ECG), mechanical vibration signals (PCG), and blood oxygen saturation (SpO2) are the core indicators for assessing cardiac function. Current technologies mainly collect these signals separately using devices such as Holter monitors, stethoscopes, and pulse oximeters. However, single-modal signals can only reflect local characteristics of cardiac activity and are insufficient to meet the comprehensive analysis needs of complex pathological mechanisms.

[0003] Current cardiac monitoring technologies face significant bottlenecks: On the one hand, traditional methods rely on single-modal signals (e.g., analyzing only ECG or PCG), neglecting the dynamic coupling relationship between electrophysiological and oxygenation signals. For example, they cannot simultaneously capture the temporal correlation between the ECG R wave and the S1 component of heart sounds. On the other hand, deep learning-based methods have inherent limitations. For instance, fixed-scale convolutional kernels cannot adapt to the multi-scale characteristics of high-frequency transient components of heart sound signals and low-frequency rhythms of ECG signals, while RNN architectures are prone to gradient vanishing when modeling long sequences, and unidirectional temporal modeling struggles to analyze the reverse physiological conduction mechanism of the cardiac cycle. Furthermore, multimodal fusion often employs feature concatenation or weighted averaging, resulting in the failure to effectively uncover deep interactions between signals (e.g., the synergistic judgment of myocardial ischemia by SpO2).

[0004] Existing technologies suffer from four major defects: (1) single signal analysis severs the spatiotemporal correlation of cardiac electromechanical activity; (2) fixed convolutional kernels limit the ability to extract multi-scale features; (3) RNN architecture is insufficient for modeling long sequences and ignores inverse temporal dependencies; and (4) the multimodal feature fusion method is simple and cannot achieve deep interaction of physiological correlations. These problems lead to insufficient sensitivity and specificity in the early diagnosis of heart diseases, especially for occult lesions such as coronary heart disease and arrhythmia, which have a high rate of missed detection. There is an urgent need for an innovative monitoring method that integrates multimodal signals, adaptive feature extraction, and bidirectional temporal modeling. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a flexible wearable cardiac function monitoring method, system and device based on multimodality, in order to solve the above-mentioned problems described in the prior art.

[0006] A first aspect of the present invention is to provide a flexible wearable method for monitoring cardiac function based on multimodality, the method comprising:

[0007] After the pre-set flexible patch sensing module is attached and placed in the corresponding area of ​​the patient's heart, multimodal signals are detected through the flexible patch sensing module; wherein, the flexible patch sensing module includes an electrocardiogram sensor, a pulse wave sensor, a heart sound sensor, a strain sensor and a microfluidic biosensor;

[0008] Acquire electrocardiogram (ECG) signals, pulse wave signals, heart sound signals, strain signals, and biological signals, and perform denoising, filtering, and normalization processing on the ECG signals, pulse wave signals, heart sound signals, strain signals, and biological signals;

[0009] The preprocessed electrocardiogram signal, pulse wave signal, heart sound signal, strain signal and biological signal are extracted using a multi-scale convolutional neural model to obtain the corresponding feature signals and perform feature fusion to output the fused signal.

[0010] Based on the fused signal, a bidirectional long short-term memory network is used to predict heart disease and classify disease types, and output assessment data corresponding to the patient's cardiac function according to the classification results.

[0011] According to one aspect of the above technical solution, the steps of extracting features from the preprocessed electrocardiogram signal, pulse wave signal, heart sound signal, strain signal, and biological signal using a multi-scale convolutional neural model to obtain corresponding feature signals, performing feature fusion, and outputting a fused signal include:

[0012] Feature extraction was performed on the electrocardiogram (ECG) signal, pulse wave signal, heart sound signal, strain signal, and biological signal at different time scales using a multi-scale convolutional neural model, resulting in ECG feature signals, pulse wave feature signals, heart sound feature signals, strain feature signals, and biometric feature signals, respectively.

[0013] At the end of the multi-scale convolutional neural model, the electrocardiogram feature signal, pulse wave feature signal, heart sound feature signal, strain feature signal and biometric feature signal are fused to integrate and superimpose all the extracted feature signals to obtain a fused signal.

[0014] According to one aspect of the above technical solution, when extracting features from the electrocardiogram (ECG) signal using a multi-scale convolutional neural model, ECG feature signals are extracted from the ECG signal through three parallel convolutional layers.

[0015] The parallel convolutional layer includes:

[0016] The first convolutional layer is used to capture local heartbeat abrupt changes based on the electrocardiogram signal;

[0017] The second convolutional layer is used to expand the receptive field based on the local heartbeat mutation features;

[0018] The third convolutional layer is used for downsampling to extract macroscopic features based on the local heartbeat mutation features, thereby obtaining the electrocardiogram feature signal;

[0019] Furthermore, each of the parallel convolutional layers is followed by a normalization layer, a batch normalization layer, an activation layer, and an attention layer.

[0020] According to one aspect of the above technical solution, when extracting features from the heart sound signal using a multi-scale convolutional neural model, a depth-separable convolutional layer is used to extract the heart sound feature signals from the heart sound signal;

[0021] The depth-separable convolutional layer includes:

[0022] A first depth separable convolutional layer is used for main frequency band analysis based on the heart sound signal;

[0023] The second depth separable convolutional layer is used for harmonic analysis based on the dominant frequency band of the heart sound signal to capture the rise of heart murmurs;

[0024] The third deep separable convolutional layer is used for broadband mode analysis to receive all abnormal sounds in the heart sound signal;

[0025] Furthermore, each layer of the depth-separable convolutional layer is followed by frequency domain windowing and absolute value activation.

[0026] According to one aspect of the above technical solution, at the end of the multi-scale convolutional neural model, the step of feature fusion of the electrocardiogram feature signal, pulse wave feature signal, heart sound feature signal, strain feature signal, and biometric feature signal to integrate and superimpose all extracted feature signals to obtain a fused signal includes at least the following:

[0027] Max pooling is performed on the time-domain features of the electrocardiogram feature signals, and key points are taken at preset time intervals;

[0028] Linear interpolation is performed on the cardiac frequency domain features of the cardiac sound feature signal to adjust the time sequence length, the auscultation recording is lengthened to the same duration as the time sequence length, and the blood oxygen protection signal is downsampled to 1 / 8 of the frequency of the ECG feature signal and scaled to 90-100% range.

[0029] Using ECG time-domain features as the query, ECG audio-domain features as the key, and blood oxygen saturation as the value, an attention weight matrix is ​​calculated to achieve weighted fusion.

[0030] The original feature signals of each signal are added to the fused feature signal to achieve residual superposition.

[0031] According to one aspect of the above technical solution, the steps of predicting heart disease and classifying disease types using a bidirectional long short-term memory network based on the fused signal, and outputting assessment data corresponding to the patient's cardiac function according to the classification results, include:

[0032] Based on the fused signal, the temporal features output by the bidirectional long short-term memory network are compressed by attention pooling through the fully connected layer of the multi-scale convolutional neural model, thereby enabling the prediction of heart disease and the classification of disease types.

[0033] Output assessment data corresponding to the patient's cardiac function based on the classification results.

[0034] According to one aspect of the above technical solution, the method further includes:

[0035] The heart sound signal and the electrocardiogram signal are segmented and processed to obtain the corresponding heart sound feature signal and electrocardiogram feature signal and then preprocessed.

[0036] Blood pressure is predicted using a pre-trained blood pressure prediction model based on the heart sound characteristic signal and the electrocardiogram characteristic signal.

[0037] A second aspect of the present invention is to provide a multimodal flexible wearable cardiac function monitoring system, applied to the method described in the above-mentioned technical solution, the system comprising:

[0038] The signal detection module is used to detect multimodal signals through the flexible patch sensing module after the preset flexible patch sensing module is attached and placed in the corresponding area of ​​the patient's heart; wherein, the flexible patch sensing module includes an electrocardiogram sensor, a pulse wave sensor, a heart sound sensor, a strain sensor and a microfluidic biosensor.

[0039] The signal processing module is used to acquire electrocardiogram signals, pulse wave signals, heart sound signals, strain signals and biological signals, and to perform noise reduction, filtering and normalization processing on the electrocardiogram signals, pulse wave signals, heart sound signals, strain signals and biological signals.

[0040] The signal fusion module is used to extract features from the preprocessed electrocardiogram signal, pulse wave signal, heart sound signal, strain signal and biological signal through a multi-scale convolutional neural model, obtain the corresponding feature signals, perform feature fusion, and output the fused signal.

[0041] The diagnostic classification module is used to predict heart disease and classify disease types based on the fused signal through a bidirectional long short-term memory network, and output assessment data corresponding to the patient's cardiac function according to the classification results.

[0042] A third aspect of the present invention is to provide a flexible wearable cardiac function monitoring device based on multimodality, applicable to the method described in the above-mentioned technical solution, the device comprising:

[0043] Flexible patch;

[0044] An electrocardiogram (ECG) sensor, including a first ECG electrode and a second ECG electrode, is disposed on the same surface of the flexible patch and spaced apart from each other.

[0045] A heart sound sensor is disposed on the flexible patch, and the heart sound sensor is disposed between the first electrocardiogram electrode and the second electrocardiogram electrode.

[0046] According to one aspect of the above technical solution, the device further includes:

[0047] A pulse wave sensor, disposed on the flexible patch, is used to detect changes in weak light signals to output pulse wave signals;

[0048] A strain sensor, mounted on the flexible patch, monitors minute deformations of the chest based on the linear relationship between the rate of change of resistance and the strain signal.

[0049] And microfluidic biosensors, which integrate microfluidic channels and immune sensors to detect biomarkers.

[0050] Compared with existing technologies, the advantages of using the multimodal-based flexible wearable cardiac function monitoring method, system, and device shown in this invention are as follows:

[0051] This invention comprehensively overcomes the technical bottlenecks of traditional cardiac monitoring by integrating multimodal sensing and deep learning. A flexible patch simultaneously acquires five-dimensional signals: electrocardiogram (ECG), pulse wave, heart sounds, myocardial deformation, and biomarkers, achieving holistic fusion analysis of cardiac electrophysiological activity, hemodynamics, mechanical vibration, tissue mechanics, and biochemical indicators. An adaptive multi-scale convolutional neural network dynamically extracts features from different frequency domains, overcoming the sensing limitations of fixed convolutional kernels. Based on the deep fusion temporal correlation characteristics of a bidirectional LSTM network, collaborative modeling using pulse wave propagation time, heart sound components, and myocardial strain data significantly improves the ability to identify pathological states. An innovative microfluidic biosensor is introduced to achieve cross-validation of sweat biomarkers and physical signals, providing early warning for occult lesions. This technology significantly improves the sensitivity and specificity of cardiac disease diagnosis, supports non-invasive, continuous dynamic monitoring, and constructs a closed-loop solution for cardiovascular health management. Attached Figure Description

[0052] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0053] Figure 1This is a flowchart illustrating a multimodal flexible wearable cardiac function monitoring method according to an embodiment of the present invention.

[0054] Figure 2 This is a structural block diagram of a flexible wearable cardiac function monitoring system based on multimodality in one embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the present invention will be more thorough and complete.

[0056] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0058] Example 1

[0059] Please see Figure 1 The first embodiment of the present invention provides a flexible wearable cardiac function monitoring method based on multimodality, the method comprising steps S10-S40:

[0060] Step S10: After attaching the preset flexible patch sensing module to the corresponding area of ​​the patient's heart, multimodal signals are detected through the flexible patch sensing module.

[0061] The flexible patch sensing module includes an electrocardiogram sensor, a pulse wave sensor, a heart sound sensor, a strain sensor, and a microfluidic biosensor.

[0062] This embodiment implements the above method using a multimodal flexible wearable cardiac function monitoring device. The device includes a flexible patch sensing module, which includes a flexible patch, an electrocardiogram sensor, a pulse wave sensor, a heart sound sensor, a strain sensor, and a microfluidic biosensor.

[0063] The electrocardiogram (ECG) sensor includes a first ECG electrode and a second ECG electrode, which are disposed on the same surface of the flexible patch and spaced apart. The heart sound sensor is disposed on the flexible patch and located between the first ECG electrode and the second ECG electrode.

[0064] More specifically, the electrocardiogram (RCG) sensor uses flexible dry electrode materials, preferably graphene electrodes. Utilizing their excellent conductivity and flexibility, it measures the electrical signals of the heart on the patient's skin surface, capturing the depolarization and repolarization processes of the heart, providing core evidence for diagnosing diseases such as arrhythmias and myocardial ischemia. The pulse wave sensor, also known as a photoplethysmography (PPG) sensor, includes a green LED and a photodiode. Based on the principles of light reflection and absorption, it acquires pulse wave signals to measure heart rate and blood oxygen saturation, and estimates blood pressure using pulse wave conduction time (PTT). The heart sound sensor is a MEMS piezoresistive heart sound sensor used to capture the mechanical vibration signals generated by the opening and closing of heart valves and the contraction and relaxation of myocardium. By analyzing the components of heart sounds, it assists in the diagnosis of valvular diseases and heart failure. The strain sensor uses flexible nanomaterials, such as strain gauges made of carbon nanotube / PDMS composite materials, which are fitted to the patient's chest skin to monitor minute deformations of the chest during heartbeats, further assessing cardiac contractility and pumping function. Microfluidic biosensors integrate microfluidic channels and biometric elements, such as immune sensors, to detect biomarkers such as cardiac troponin I (cTnI) and brain natriuretic peptide (BNP) through skin microexudates, enabling early warning of heart diseases.

[0065] When using this flexible wearable cardiac function monitoring device based on multimodality, a flexible patch is attached to the corresponding area of ​​the patient's heart. Then, based on the sensor integrated in the flexible patch, multimodal signals are output, including the detection of the patient's corresponding electrocardiogram signal, pulse wave signal, heart sound signal, strain signal and biological signal. Thus, multiple signals can be acquired simultaneously through the flexible wearable cardiac function monitoring device, which is especially suitable for long-term dynamic monitoring of bedridden patients and reduces the inconvenience of use for patients.

[0066] Step S20: Acquire electrocardiogram (ECG) signals, pulse wave signals, heart sound signals, strain signals, and biological signals, and perform noise reduction, filtering, and normalization processing on the ECG signals, pulse wave signals, heart sound signals, strain signals, and biological signals.

[0067] In this embodiment, after the multimodal signals are detected by the flexible patch sensing module, in order to facilitate subsequent disease prediction and type classification, the electrocardiogram signal, pulse wave signal, heart sound signal, strain signal and biological signal will be denoised, filtered and normalized.

[0068] Among them, the electrocardiogram (ECG) sensor records the potential difference generated by the heart's electrical activity through contact with the skin via electrodes, forming an ECG. By analyzing the waveform characteristics of P waves, QRS waves, and T waves, abnormalities such as arrhythmias and myocardial infarction can be identified. The pulse wave sensor uses a green LED to illuminate the skin. Because the absorption of green light by hemoglobin changes with blood vessel volume, a photodiode receives the reflected light signal and converts it into an electrical signal. Algorithms then extract heart rate, blood oxygen saturation, and pulse wave characteristic parameters. The phonocardiogram sensor converts the mechanical vibrations of the heart into electrical signals, collecting the first heart sound S1, the second heart sound S2, and additional heart sounds such as the third heart sound S3 and the fourth heart sound S4. Combined with time-frequency analysis technology, abnormal heart sound patterns are identified. The strain sensor changes its resistance value with the deformation of chest muscles and skin, converting it into a voltage signal through a Wheatstone bridge to reflect the mechanical changes during heart contraction and assist in assessing cardiac function. The microfluidic biosensor utilizes the reaction between skin microexudates and biorecognition elements, such as antibody-antigen specific binding. Through electrochemical or optical detection signals, the concentration of biomarkers is quantified to predict the risk of myocardial injury.

[0069] Specifically, the above signals are denoised, filtered and normalized, including synchronously transmitting the electrocardiogram signal, pulse wave signal, heart sound signal, strain signal and biological signal to the microprocessor, such as synchronously transmitting to the low power ARM chip, and then aligning the above electrocardiogram signal, pulse wave signal, heart sound signal, strain signal and biological signal based on time series to construct a multimodal dataset.

[0070] Step S30: The preprocessed electrocardiogram signal, pulse wave signal, heart sound signal, strain signal and biological signal are extracted using a multi-scale convolutional neural model to obtain the corresponding feature signals and perform feature fusion to output the fused signal.

[0071] In this embodiment, after preprocessing the above-mentioned electrocardiogram signals, pulse wave signals, heart sound signals, strain signals and biological signals and constructing a multimodal dataset, the above-mentioned electrocardiogram signals, pulse wave signals, heart sound signals, strain signals and biological signals are respectively extracted by a multi-scale convolutional neural model to obtain the corresponding feature signals and perform feature fusion to obtain the fused signal.

[0072] The steps involved in extracting features from the preprocessed electrocardiogram (ECG), pulse wave, heart sound, strain, and biological signals using a multi-scale convolutional neural model, obtaining corresponding feature signals, fusing these features, and outputting the fused signal include:

[0073] Feature extraction was performed on the electrocardiogram (ECG) signal, pulse wave signal, heart sound signal, strain signal, and biological signal at different time scales using a multi-scale convolutional neural model, resulting in ECG feature signals, pulse wave feature signals, heart sound feature signals, strain feature signals, and biometric feature signals, respectively.

[0074] At the end of the multi-scale convolutional neural model, the electrocardiogram feature signal, pulse wave feature signal, heart sound feature signal, strain feature signal and biometric feature signal are fused to integrate and superimpose all the extracted feature signals to obtain a fused signal.

[0075] Specifically, when extracting features from the electrocardiogram (ECG) signal using a multi-scale convolutional neural model, ECG feature signals are extracted from the ECG signal through three parallel convolutional layers.

[0076] The parallel convolutional layer includes:

[0077] The first convolutional layer is used to capture local heartbeat abrupt changes based on the electrocardiogram signal;

[0078] The second convolutional layer is used to expand the receptive field based on the local heartbeat mutation features;

[0079] The third convolutional layer is used for downsampling to extract macroscopic features based on the local heartbeat mutation features, thereby obtaining the electrocardiogram feature signal;

[0080] Furthermore, each of the parallel convolutional layers is followed by a normalization layer, a batch normalization layer, an activation layer, and an attention layer.

[0081] Specifically, when extracting features from the heart sound signal using a multi-scale convolutional neural model, a depth-separable convolutional layer is used to extract the heart sound feature signals from the heart sound signal.

[0082] The depth-separable convolutional layer includes:

[0083] A first depth separable convolutional layer is used for main frequency band analysis based on the heart sound signal;

[0084] The second depth separable convolutional layer is used for harmonic analysis based on the dominant frequency band of the heart sound signal to capture the rise of heart murmurs;

[0085] The third deep separable convolutional layer is used for broadband mode analysis to receive all abnormal sounds in the heart sound signal;

[0086] Furthermore, each layer of the depth-separable convolutional layer is followed by frequency domain windowing and absolute value activation.

[0087] The step of fusing the electrocardiogram (ECG) feature signals, pulse wave feature signals, heart sound feature signals, strain feature signals, and biometric feature signals at the end of the multi-scale convolutional neural model to integrate and superimpose all extracted feature signals to obtain a fused signal includes at least the following:

[0088] Max pooling is performed on the time-domain features of the electrocardiogram feature signals, and key points are taken at preset time intervals;

[0089] Linear interpolation is performed on the cardiac frequency domain features of the cardiac sound feature signal to adjust the time sequence length, the auscultation recording is lengthened to the same duration as the time sequence length, and the blood oxygen protection signal is downsampled to 1 / 8 of the frequency of the ECG feature signal and scaled to 90-100% range.

[0090] Using ECG time-domain features as the query, ECG audio-domain features as the key, and blood oxygen saturation as the value, an attention weight matrix is ​​calculated to achieve weighted fusion.

[0091] The original feature signals of each signal are added to the fused feature signal to achieve residual superposition.

[0092] Specifically, feature extraction is performed on the aforementioned signals, that is, extracting the deep-seated features implicit within the signals to better utilize them for subsequent disease prediction and classification. For electrocardiogram (ECG) signals, this includes calculating parameters such as the RR interval, ST segment shift, and QT interval to extract corresponding ECG feature signals. For pulse wave signals, this includes identifying the pulse wave conduction velocity and waveform characteristics to extract corresponding pulse wave feature signals. For heart sound signals, this includes identifying the frequency, intensity, and duration of heart sound components to extract corresponding heart sound feature signals. For strain signals, this includes calculating the amplitude and frequency of chest deformation to extract corresponding strain feature signals. Finally, for biological signals, this includes comparing biomarkers with normal thresholds to determine abnormal concentrations, thereby extracting corresponding biological feature signals.

[0093] In this embodiment, the electrocardiogram characteristic signal, pulse wave characteristic signal, heart sound characteristic signal, strain characteristic signal and biological characteristic signal mentioned above are fused together to obtain a fused signal.

[0094] Step S40: Based on the fused signal, predict heart disease and classify disease types through a bidirectional long short-term memory network, and output assessment data corresponding to the patient's heart function according to the classification results.

[0095] In this embodiment, based on the fused signal, a bidirectional long short-term memory network is used to predict heart disease and classify disease types, that is, to determine whether heart disease exists and to classify the types of heart disease, so as to output assessment data corresponding to the patient's cardiac function based on the classification results.

[0096] Specifically, a bidirectional long short-term memory network is used, and the model is trained based on a large amount of clinical data to achieve functions such as arrhythmia classification, atrial fibrillation, premature ventricular contractions, myocardial ischemia early warning, and heart failure risk assessment.

[0097] More specifically, the output assessment data can be transmitted to smart terminals or cloud platforms via Bluetooth or Low Energy Wide Area Network (LEW). These smart terminals can be used by patients or medical staff. On the patient's end, real-time data such as heart rate, blood pressure, heart sound waveforms, and biomarker concentrations can be displayed, along with a risk assessment report. On the medical staff's end, remote monitoring can be performed, and AI-powered diagnostic suggestions can be combined to assist clinical decision-making.

[0098] In summary, compared with existing technologies, the multimodal flexible wearable cardiac function monitoring method shown in this embodiment has the following advantages:

[0099] This embodiment comprehensively overcomes the technical bottlenecks of traditional cardiac monitoring by integrating multimodal sensing and deep learning. A flexible patch simultaneously acquires five-dimensional signals including electrocardiogram, pulse wave, heart sounds, myocardial deformation, and biomarkers, achieving holistic fusion analysis of cardiac electrophysiological activity, hemodynamics, mechanical vibration, tissue mechanics, and biochemical indicators. An adaptive multi-scale convolutional neural network dynamically extracts features from different frequency domains, overcoming the perception limitations of fixed convolutional kernels. Based on the deep fusion temporal correlation characteristics of a bidirectional LSTM network, collaborative modeling using pulse wave propagation time, heart sound components, and myocardial strain data significantly improves the ability to identify pathological states. An innovative microfluidic biosensor is introduced to achieve cross-validation of sweat biomarkers and physical signals, providing early warning for occult lesions. This technology significantly improves the sensitivity and specificity of cardiac disease diagnosis, supports non-invasive, continuous dynamic monitoring, and constructs a closed-loop solution for cardiovascular health management.

[0100] Example 2

[0101] The second embodiment of the present invention provides a flexible wearable cardiac function monitoring method based on multimodality. The method shown in this embodiment is basically similar to the method shown in the first embodiment, except that:

[0102] In this embodiment, the steps of predicting heart disease and classifying disease types using a bidirectional long short-term memory network based on the fused signal, and outputting assessment data corresponding to the patient's cardiac function according to the classification results, include:

[0103] Based on the fused signal, the temporal features output by the bidirectional long short-term memory network are compressed by attention pooling through the fully connected layer of the multi-scale convolutional neural model, thereby enabling the prediction of heart disease and the classification of disease types.

[0104] Output assessment data corresponding to the patient's cardiac function based on the classification results.

[0105] Specifically, the temporal feature matrix (dimension: time step T × feature dimension D) output by the Bidirectional Long Short-Term Memory Network (Bi-LSTM) is input into a fully connected layer, where a nonlinear transformation is performed: h'_t = ReLU(W_fc·h_t + b_fc), where W_fc is the weight matrix. This maps the original features to a 128-dimensional high-discrimination space, enhancing the expressive power of pathological features such as ST segment shift and heart sound splitting. Then, attention weights are calculated for the feature vector h'_t at each time step, and the weighted aggregated fixed-dimensional vector is used as the classification input: h_att = Σ(α_t·h'_t). Finally, the compressed feature vector h_att is input into the fully connected classifier, outputting disease probability predictions and clinical assessment data such as the NYHA functional classification.

[0106] In addition, the method also includes:

[0107] The heart sound signal and the electrocardiogram signal are segmented and processed to obtain the corresponding heart sound feature signal and electrocardiogram feature signal and then preprocessed.

[0108] Blood pressure is predicted using a pre-trained blood pressure prediction model based on the heart sound characteristic signal and the electrocardiogram characteristic signal.

[0109] The blood pressure prediction model is a cross-modal regression network based on LSTM-Transformer.

[0110] Specifically, when segmenting the electrocardiogram (ECG) signal, the peak value of the R wave is used as the anchor point, and a window of [R-100ms, R+400ms] is extracted to cover the complete QRS-T cycle, and the QRS duration and ST segment slope are extracted. When segmenting the heart sound signal (PCG), the R wave of the ECG is locked synchronously, and the [systolic and diastolic] phases are divided according to the S1 / S2 components, and the S1 amplitude and S2-S1 interval are extracted.

[0111] This embodiment improves the sensitivity and specificity of disease diagnosis by performing attention-enhanced classification and ECG / PCG fusion blood pressure prediction, while retaining the advantages of multimodal signals. It also adds non-invasive continuous blood pressure monitoring function, providing a closed-loop management solution for cardiovascular diseases.

[0112] Example 3

[0113] Please see Figure 2 A third embodiment of the present invention provides a multimodal flexible wearable cardiac function monitoring system, applied to the method described in any of the above embodiments, the system comprising:

[0114] The signal detection module 10 is used to detect multimodal signals through the flexible patch sensing module after the preset flexible patch sensing module is attached and placed in the corresponding area of ​​the patient's heart. The flexible patch sensing module includes an electrocardiogram sensor, a pulse wave sensor, a heart sound sensor, a strain sensor, and a microfluidic biosensor.

[0115] The signal processing module 20 is used to acquire electrocardiogram signals, pulse wave signals, heart sound signals, strain signals and biological signals, and to perform noise reduction, filtering and normalization processing on the electrocardiogram signals, pulse wave signals, heart sound signals, strain signals and biological signals.

[0116] The signal fusion module 30 is used to extract features from the preprocessed electrocardiogram signal, pulse wave signal, heart sound signal, strain signal and biological signal through a multi-scale convolutional neural model, obtain the corresponding feature signals, perform feature fusion, and output the fused signal.

[0117] The diagnostic classification module 40 is used to predict heart disease and classify disease types based on the fused signal through a bidirectional long short-term memory network, and output assessment data corresponding to the patient's cardiac function according to the classification results.

[0118] In summary, compared with existing technologies, the advantages of using the multimodal flexible wearable cardiac function monitoring system shown in this embodiment are as follows:

[0119] This embodiment comprehensively overcomes the technical bottlenecks of traditional cardiac monitoring by integrating multimodal sensing and deep learning. A flexible patch simultaneously acquires five-dimensional signals including electrocardiogram, pulse wave, heart sounds, myocardial deformation, and biomarkers, achieving holistic fusion analysis of cardiac electrophysiological activity, hemodynamics, mechanical vibration, tissue mechanics, and biochemical indicators. An adaptive multi-scale convolutional neural network dynamically extracts features from different frequency domains, overcoming the perception limitations of fixed convolutional kernels. Based on the deep fusion temporal correlation characteristics of a bidirectional LSTM network, collaborative modeling using pulse wave propagation time, heart sound components, and myocardial strain data significantly improves the ability to identify pathological states. An innovative microfluidic biosensor is introduced to achieve cross-validation of sweat biomarkers and physical signals, providing early warning for occult lesions. This technology significantly improves the sensitivity and specificity of cardiac disease diagnosis, supports non-invasive, continuous dynamic monitoring, and constructs a closed-loop solution for cardiovascular health management.

[0120] Example 4

[0121] A fourth embodiment of the present invention provides a multimodal flexible wearable cardiac function monitoring device, applied to the method described in any of the above embodiments, the device comprising:

[0122] Flexible patch;

[0123] An electrocardiogram (ECG) sensor, including a first ECG electrode and a second ECG electrode, is disposed on the same surface of the flexible patch and spaced apart from each other.

[0124] A heart sound sensor is disposed on the flexible patch, and the heart sound sensor is disposed between the first electrocardiogram electrode and the second electrocardiogram electrode.

[0125] The device further includes:

[0126] A pulse wave sensor, disposed on the flexible patch, is used to detect changes in weak light signals to output pulse wave signals;

[0127] A strain sensor, mounted on the flexible patch, monitors minute deformations of the chest based on the linear relationship between the rate of change of resistance and the strain signal.

[0128] And microfluidic biosensors, which integrate microfluidic channels and immune sensors to detect biomarkers.

[0129] In this embodiment, a flexible patch simultaneously acquires five-dimensional signals including electrocardiogram, pulse wave, heart sound, myocardial deformation, and biomarkers, achieving holistic fusion analysis of cardiac electrophysiological activity, hemodynamics, mechanical vibration, tissue mechanics, and biochemical indicators. An adaptive multi-scale convolutional neural network dynamically extracts features from different frequency domains, overcoming the perception limitations of fixed convolutional kernels. Based on the deep fusion temporal correlation characteristics of a bidirectional LSTM network, collaborative modeling using pulse wave conduction time, heart sound components, and myocardial strain data significantly improves the ability to identify pathological states. An innovative microfluidic biosensor is introduced to achieve cross-validation of sweat biomarkers and physical signals, providing early warning for occult lesions. This technology significantly improves the sensitivity and specificity of cardiac disease diagnosis, supports non-invasive, continuous dynamic monitoring, and constructs a closed-loop solution for cardiovascular health management.

[0130] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequential list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0132] More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable storage media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0133] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0134] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0135] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A multi-modal based flexible wearable cardiac function monitoring system, characterized in that, The system comprises: a signal detection module, configured to detect multi-modal signals through a flexible patch sensor module after the flexible patch sensor module is arranged on a corresponding region of a patient's heart; a signal processing module, configured to acquire electrocardio signals, pulse wave signals, heart sound signals, strain signals and biological signals, and to perform denoising, filtering and normalization processing on the electrocardio signals, pulse wave signals, heart sound signals, strain signals and biological signals; a signal fusion module, configured to extract features from the preprocessed electrocardio signals, pulse wave signals, heart sound signals, strain signals and biological signals through a multi-scale convolutional neural model, to obtain corresponding feature signals and perform feature fusion, and to output a fusion signal; a diagnosis classification module, configured to predict heart diseases and classify disease types through a bidirectional long short-term memory network according to the fusion signal, and to output evaluation data corresponding to the patient's heart function according to the classification result. The system is used in the following method: detecting multi-modal signals through a flexible patch sensor module after the flexible patch sensor module is arranged on a corresponding region of a patient's heart; acquiring electrocardio signals, pulse wave signals, heart sound signals, strain signals and biological signals, and performing denoising, filtering and normalization processing on the electrocardio signals, pulse wave signals, heart sound signals, strain signals and biological signals; extracting features from the preprocessed electrocardio signals, pulse wave signals, heart sound signals, strain signals and biological signals through a multi-scale convolutional neural model, obtaining corresponding feature signals and performing feature fusion, and outputting a fusion signal; predicting heart diseases and classifying disease types through a bidirectional long short-term memory network according to the fusion signal, and outputting evaluation data corresponding to the patient's heart function according to the classification result. When extracting features from the heart sound signals through the multi-scale convolutional neural model, a depth separable convolutional layer is used to extract heart sound feature signals from the heart sound signals. The depth separable convolutional layer comprises: a first depth separable convolutional layer, configured to perform main frequency band analysis according to the heart sound signals; a second depth separable convolutional layer, configured to perform harmonic analysis according to the main frequency band of the heart sound signals, and to capture the rebound of heart murmurs; a third depth separable convolutional layer, configured to perform wide frequency mode analysis, and to receive all abnormal sounds in the heart sound signals; and a frequency domain windowing and absolute value activation are connected after each layer of the depth separable convolutional layer.

2. The multi-modal based flexible wearable cardiac function monitoring system of claim 1, wherein, The step of extracting features from the preprocessed electrocardio signals, pulse wave signals, heart sound signals, strain signals and biological signals through a multi-scale convolutional neural model, obtaining corresponding feature signals and performing feature fusion, and outputting a fusion signal, comprises: extracting features from the electrocardio signals, pulse wave signals, heart sound signals, strain signals and biological signals at different time scales through a multi-scale convolutional neural model, and obtaining electrocardio feature signals, pulse wave feature signals, heart sound feature signals, strain feature signals and biological marker feature signals, respectively. At the end of the multi-scale convolutional neural model, the electrocardiogram feature signal, the pulse wave feature signal, the heart sound feature signal, the strain feature signal and the biomarker feature signal are fused to integrate and superimpose all the extracted feature signals to obtain a fusion signal.

3. The multi-modal based flexible wearable cardiac function monitoring system of claim 2, wherein, When the multi-scale convolutional neural model is used for feature extraction of the electrocardiogram signal, three parallel convolutional layers are used to extract the electrocardiogram feature signal in the electrocardiogram signal. The parallel convolutional layers include: The first convolutional layer is used to capture local heartbeat mutation features from the electrocardiogram signal. The second convolutional layer is used to expand the receptive field based on the local heartbeat mutation features. The third convolutional layer is used to downsample to extract macro features from the local heartbeat mutation features to obtain the electrocardiogram feature signal. And after each layer of the parallel convolutional layer, a normalization layer, a batch normalization layer, an activation layer and an attention layer are connected.

4. The multi-modal based flexible wearable cardiac function monitoring system of claim 1, wherein, At the end of the multi-scale convolutional neural model, the electrocardiogram feature signal, the pulse wave feature signal, the heart sound feature signal, the strain feature signal and the biomarker feature signal are fused to integrate and superimpose all the extracted feature signals to obtain a fusion signal. The steps include: The maximum pooling is performed on the electrocardiogram time domain features of the electrocardiogram feature signal, and the key points are taken every pre-set time. The heart sound frequency domain features of the heart sound feature signal are linearly interpolated to adjust the time sequence length, the auscultation recording is lengthened to the same length of the time sequence length, the blood oxygen protection signal is downsampled to 1 / 8 frequency of the electrocardiogram feature signal and scaled to the range of 90-100%. The electrocardiogram time domain features are taken as Query, the heart sound frequency domain features are taken as Key, and the blood oxygen saturation is taken as Value to calculate the attention weight matrix and realize weighted fusion. The original feature signals and the fusion feature signals of each signal are added to realize residual superposition.

5. The multi-modal based flexible wearable cardiac function monitoring system of claim 1, wherein, According to the fusion signal, the bidirectional long short-term memory network is used to predict the heart disease and classify the disease type, and the evaluation data corresponding to the patient's heart function is output according to the classification result. According to the fusion signal, the time sequence features output by the bidirectional long short-term memory network are attention-pooled and compressed through the fully connected layer of the multi-scale convolutional neural model to realize the prediction of the heart disease and the classification of the disease type. The evaluation data corresponding to the patient's heart function is output according to the classification result.

6. The multi-modal based flexible wearable cardiac function monitoring system according to any one of claims 1-5, wherein, The system is also used for the following method: The heart sound signal and the electrocardiogram signal are segmented to obtain corresponding heart sound feature signals and electrocardiogram feature signals and to pre-process them; The blood pressure prediction model is pre-trained to predict blood pressure based on the heart sound feature signals and the electrocardiogram feature signals.

7. A multi-modal based flexible wearable cardiac function monitoring device, characterized in that, The device is applied to the system of any one of claims 1-6, and the device comprises: A flexible patch; An electrocardiogram sensor comprising a first electrocardiogram electrode and a second electrocardiogram electrode, which are arranged on the same surface of the flexible patch and are spaced apart; A heart sound sensor arranged on the flexible patch, and the heart sound sensor is arranged between the first electrocardiogram electrode and the second electrocardiogram electrode.

8. The multi-modal based flexible wearable cardiac function monitoring device of claim 7, wherein, The device further comprises: A pulse wave sensor is arranged on the flexible patch to detect weak light signal changes to output a pulse wave signal; A strain sensor is arranged on the flexible patch to monitor chest micro-deformation according to a linear relationship between a resistance change rate and a strain signal; And a microfluidic biosensor integrates a microfluidic channel and an immunosensor to detect biomarkers.

Images