A method and system for processing synchronous heart sound electrocardiogram signals.

Abstract

This invention discloses a method and system for processing synchronous heart sound and electrocardiogram (ECG) signals. The method includes: synchronously acquiring heart sound and ECG signals; segmenting the signals into several first heart sound and ECG signal segments; preprocessing and normalizing these segments to obtain second heart sound and ECG signal segments; splicing the second heart sound and ECG signal segments to obtain a one-dimensional heart sound and ECG signal, and extracting a first feature vector; converting the second heart sound and ECG signal segments into two-dimensional heart sound and ECG waveform images respectively, splicing them to obtain a two-dimensional heart sound and ECG waveform image, and extracting a second feature vector; splicing the first and second feature vectors to obtain a third feature vector; classifying the signals using the sigmoid function based on the third feature vector; and counting the abnormality based on the classification results, determining an abnormality when the abnormality rate exceeds a threshold. This invention can improve the accuracy and robustness of signal processing, providing more accurate diagnostic information for heart failure clinical practice and reducing potential life-threatening risks.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of heart sound and electrocardiogram signal processing, and particularly relates to a processing method and system for homologous and synchronous heart sound and electrocardiogram signals. BACKGROUND

[0002] Heart failure, also known as "heart muscle failure" (HF, Heart Failure), is caused by abnormality of heart structure or function, which leads to dysfunction of heart ventricular systolic and diastolic function, and the heart cannot pump out blood corresponding to venous return and body tissue metabolism. Left ventricular dysfunction is a more serious and common type of heart failure. In clinical diagnosis, echocardiography can diagnose the problems of heart contraction and diastolic function by checking the shape of each atrioventricular cavity and the problem of heart valve, so as to diagnose whether there is heart failure, and thus reflect the severity of heart failure, but it is difficult to popularize to daily home monitoring.

[0003] Electrocardiogram and heart sound reflect the electrical activity and mechanical activity of the heart respectively, and the waveforms of electrocardiogram signal and heart sound signal are formed by the total potential change in the cardiac muscle and repolarization process, and the sound of heart beating. In particular, non-invasive detection methods can greatly reduce the economic burden of patients. Therefore, heart sound and electrocardiogram monitoring and diagnosis are important technical means for effective screening and follow-up of heart failure. Practice shows that it is far from enough to use only one of electrocardiogram or heart sound signal to detect heart failure, because some patients do not show abnormality in electrocardiogram information but show abnormality in heart sound, and vice versa.

[0004] Homologous and synchronous heart sound and electrocardiogram are used to calculate physiological parameters of the heart for diagnosing heart function, including important parameters such as electromechanical activation time EMAT (Electromechanical Activation Time), systolic dysfunction index SDI (Systolic Dysfunction Index), left ventricular systolic time LVST (Left Ventricular Systolic Time), ratio of EMAT to LVST, ratio of EMAT to RR interval (EMAT / RR), etc. For example, the time interval EMAT value between the starting point of QRS wave of electrocardiogram and the first peak value of first heart sound S1 of heart sound, which is confirmed by previous clinical practice, indicates that when the EMAT value is prolonged, the left ventricular systolic function is reduced, and the mitral valve closing time is prolonged. EMAT and EMAT / RR have important value in diagnosing chronic heart failure. Practice shows that there is a complementary relationship between heart sound and electrocardiogram dual-mode signals and images.

[0005] CN115640507A, published on January 24, 2023, entitled "Abnormal number screening method based on electrocardio-acoustic joint analysis", comprising: constructing an electrocardio segmentation model and an acoustic segmentation model respectively, using the segmentation model to determine the state to which each frame of electrocardio data and acoustic data belongs; input the synchronous electrocardio data and acoustic data segmentation results into the corresponding confidence module, and take the synchronous electrocardio data and acoustic data with the highest confidence as the effective signal segment; decode the effective signal segment, and calculate the electrocardio-acoustic joint discrete feature using the decoded effective signal segment; fuse the synchronous electrocardio data, acoustic data and electrocardio-acoustic joint discrete feature in the decoded effective signal segment, input the fused data into the prior distribution network, and input the output data of the prior distribution network into the screening module to screen out abnormal electrocardio and acoustic data, and reduce the interference of noise on abnormal data screening as much as possible.

[0006] CN116019480A, published on April 28, 2023, entitled "Three-valve stenosis identification method and device fusing acoustic-electrocardio signal time sequence features", the main steps are: step 1, synchronously collect the acoustic cardiogram and electrocardiogram of the detected person, and set the sampling frequency to 1000Hz; step 2, based on the electrocardiogram collected in step 1, the feature of the electrocardio signal is positioned, and the peak position of the electrocardio features R wave, T wave and P wave is determined; step 3, based on the acoustic cardiogram collected in step 1, the feature of the acoustic signal is positioned, and the peak position of the first acoustic S1 and the second acoustic S2 is determined; step 4, in the same cardiac cycle, the time difference t1 between the electrocardio feature R wave and the first acoustic S1 and the time difference t2 between the electrocardio feature T wave and the second acoustic S2 are obtained, if the time interval of t1 and t2 is abnormal, then continue to diagnose three-valve stenosis, otherwise return normal result; step 5, if three-valve stenosis diagnosis is performed, the amplitudes of the diastolic and systolic periods of the acoustic signal are sampled, if the diastolic period of the acoustic signal appears noise, it is determined to be caused by three-valve stenosis; if the systolic period of the acoustic signal appears noise, it is determined to be caused by three-valve stenosis.

[0007] CN118553421A, published on August 27, 2024, entitled "A Hypertension Population Heart Failure Prediction System Based on Heart Sound and ECG Monitoring Data", including a physiological signal acquisition module, a data transmission module, and a data processing and analysis module; the physiological signal acquisition module is used to collect the heart sound and ECG signal of the patient; the data transmission module is used to transmit the collected data to the total control end; the data processing and analysis module is used to analyze and process the collected data to predict the possibility of heart failure; by fusing the feature abstraction ability of CNN and the time series prediction ability of BI LSTM, heart failure can be effectively predicted from short-term ECG data, by introducing the cross-attention mechanism, the time and frequency domain features can be effectively fused, providing more information for heart failure prediction, and more effectively extracting useful information from time and frequency domain features for prediction tasks, improving the prediction performance and work efficiency of the model. However, the heart sound does not mention how to effectively utilize.

[0008] CN117281523A, published on December 26, 2023, entitled "ECG and Heart Sound Data Processing Method, System, Electronic Device and Storage Medium", mainly includes: the terminal processor simultaneously acquires the ECG and heart sound data of the target user through the ECG and heart sound monitoring device and acquires the body state data of the target user through the body state monitoring device in the first time period; the ECG and heart sound data includes ECG data and heart sound data; the terminal processor determines the reliability of the ECG and heart sound data according to the body state data; if the reliability exceeds the preset data value, the terminal processor calculates the first analysis strategy of the ECG and heart sound data according to the current data processing pressure of the terminal processor and the current data processing pressure of the cloud server; the first analysis strategy includes the analysis content of the terminal processor and the analysis content of the cloud server; the terminal processor uploads the analysis content of the cloud server to the cloud server; the cloud server analyzes the analysis content of the cloud server according to the determined analysis method to determine the cloud analysis result.

[0009] CN116563531A, published on August 8, 2023, entitled "Method and device for judging mitral valve prolapse and pathological segmentation based on heart ultrasound image", mainly includes: acquiring a heart ultrasound image to establish a heart ultrasound image dataset; denoising and image processing each picture of the heart ultrasound image dataset; training and testing a small sample classification model based on the denoised and preprocessed pictures; classifying existing heart ultrasound images according to the small sample classification model to obtain a heart mitral valve medical image dataset and a heart mitral valve prolapse image dataset, and performing image processing; inputting, training and verifying a pathological segmentation neural model based on the heart mitral valve prolapse image dataset; inputting the heart mitral valve medical image and the heart mitral valve prolapse image of the small sample classification model into the pathological segmentation neural model to preliminarily verify the pathological segmentation neural model, and obtaining a classification-segmentation model; inputting new medical ultrasound images into the classification-segmentation model in turn to detect whether classification and pathological segmentation are completed.

[0010] CN117322887A, published on January 2, 2024, entitled "ECG and heart sound diagnosis method, device and equipment based on artificial neural network", the method comprises: training a first neural network model based on first training data corresponding to sample data and actual data labels, and training a second neural network model based on second training data corresponding to sample data and actual data labels; until the first neural network model and the second neural network model are converged; the sample data includes user basic data and electrocardiogram and phonocardiogram obtained based on the same time frequency; obtaining first training data features corresponding to the first training data by the trained first neural network model, and obtaining second training data features corresponding to the second training data based on the trained second neural network model; training the heart recognition model according to the first training data features and the second training data features until the loss value of the heart recognition model meets the preset condition; obtaining the predicted data label corresponding to the user data of the user to be identified according to the trained heart recognition model. The above method does not clearly define the technical connotation of the first neural network model and the second neural network model.

[0011] Chinese patent CN 115177262A, published on October 14, 2022, entitled "A heart sound and electrocardiogram combined diagnosis device and system based on deep learning", according to the heart sound, electrocardiogram original signal is preprocessed to obtain each equal length segment, the CRDNet network model is adopted, the classification result corresponding to each equal length segment is calculated; the diagnostic analysis report is formed according to each classification result. Among them, the CRDNet network model includes: PCG specific modal encoder and ECG specific modal encoder with the same structure, dense fusion encoder, and collaborative decision module; each specific modal encoder is used for step-by-step extraction of features of the input equal length segment, the zeroth level is the space-time feature, and finally the deep feature is obtained, and the deep feature is used for classification to obtain the modal internal classification result under the current equal length segment input; the dense fusion encoder is used for step-by-step fusion of the features extracted by the two specific modal encoders, in each level of fusion process, the contribution of different modalities and each modality to the classification result is adaptively evaluated, a pixel-level weight map is generated for the multi-level aggregation features of each modality, the multi-level aggregation features of each modality are multiplied with the corresponding pixel-level weight map to obtain the weighted multi-level aggregation features of the modality; the multi-level aggregation features of each modality after weighting and the fusion features obtained by the previous level of fusion are fused through convolution operation to generate the fusion features of the current level, which is used to obtain the joint classification result under the current equal length segment input; the multi-level aggregation features of each modality used to generate the fusion features of the zeroth level are the space-time features corresponding to the modality, and the multi-level aggregation features of each modality used to generate the fusion features of other levels are obtained by aggregating the current level features and the previous level features extracted by the specific modal encoder of the modality; the collaborative decision module is used for weighted addition of the modal internal classification results of PCG and ECG and the joint classification result to obtain the final classification result under the current equal length segment input.

[0012] Based on the above description, taking heart failure as an example, the system and method for combined diagnosis of heart sound and electrocardiogram are still a relatively weak field, and there is a lot of room for improvement in accuracy and robustness. SUMMARY

[0013] The purpose of the present application is to provide a processing method and system for homologous and synchronous heart sound and electrocardiogram signals, which can improve the accuracy and robustness of signal processing, and the classification result obtained can provide more accurate diagnostic information for heart failure clinic and reduce potential life danger.

[0014] In order to achieve the above purpose, the solution of the present application is:

[0015] A processing method for homologous and synchronous heart sound and electrocardiogram signals, comprising,

[0016] Synchronously acquire heart sound signals and electrocardio signals in a time period, wherein the time period contains several heart cycles;

[0017] Divide the heart sound signals according to length to obtain several first heart sound signal segments; divide the electrocardio signals according to length to obtain several first electrocardio signal segments; wherein length contains at least one heart cycle;

[0018] Preprocess the first heart sound signal segments, and then normalize to obtain second heart sound signal segments; preprocess the first electrocardio signal segments, and then normalize to obtain second electrocardio signal segments;

[0019] Vertically splice the second heart sound signal segments and the second electrocardio signal segments to obtain one-dimensional heart sound electrocardio signals, and extract first feature vectors from the one-dimensional heart sound electrocardio signals using a heart sound electrocardio signal neural network;

[0020] Convert the second heart sound signal segments and the second electrocardio signal segments into heart sound two-dimensional waveform images and electrocardio two-dimensional waveform images respectively, vertically splice the heart sound two-dimensional waveform images and the electrocardio two-dimensional waveform images to obtain two-dimensional heart sound electrocardio waveform images, and extract second feature vectors from the two-dimensional heart sound electrocardio waveform images using a heart sound electrocardio image neural network;

[0021] Splice the first feature vectors and the second feature vectors to obtain third feature vectors;

[0022] Classify using Sigmoid according to the third feature vectors;

[0023] Count according to the classification results, and determine as abnormal when the abnormality ratio exceeds a threshold value;

[0024] Wherein, the one-dimensional heart sound electrocardio signals are extracted into first feature vectors using a heart sound electrocardio signal neural network, including,

[0025] Extract shallow features from the one-dimensional heart sound electrocardio signals using a first convolutional layer to obtain a shallow feature map;

[0026] Based on the shallow feature map, extract horizontal direction time sequence features and vertical direction correlation features using a coordinate attention mechanism to obtain a correlation feature map;

[0027] Based on the correlation feature map, extract deep features using a second convolutional layer to obtain a deep feature map;

[0028] Jump connect the one-dimensional heart sound electrocardio signals with the deep feature map and reduce the spatial feature dimension to obtain a spatial feature map;

[0029] The features of the space feature map are tiled and time features are extracted to obtain a first feature vector.

[0030] The heart sound signal is divided according to length to obtain a plurality of first heart sound signal segments, including

[0031] The heart sound signal is divided according to length , and the remaining part is filled or discarded if it is less than length .

[0032] The electrocardiogram signal is divided according to length to obtain a plurality of first electrocardiogram signal segments, including

[0033] The electrocardiogram signal is divided according to length , and the remaining part is filled or discarded if it is less than length .

[0034] The first heart sound signal segment is preprocessed, including

[0035] The first heart sound signal segment is sequentially subjected to Butterworth band-pass filtering and wavelet transform denoising.

[0036] The first electrocardiogram signal segment is preprocessed, including

[0037] The first electrocardiogram signal segment is sequentially subjected to Butterworth band-pass filtering and notch filtering.

[0038] Based on the shallow feature map, a coordinate attention mechanism is used to extract horizontal direction time sequence features and vertical direction correlation features to obtain a correlation feature map, including

[0039] The shallow feature map is averaged and pooled in the horizontal and vertical directions to obtain one-dimensional time sequence features and one-dimensional correlation features.

[0040] The one-dimensional correlation features are flipped and concatenated with the one-dimensional time sequence features to obtain a correlation feature map.

[0041] The heart sound two-dimensional waveform image and the electrocardiogram two-dimensional waveform image are vertically concatenated to obtain a two-dimensional heart sound electrocardiogram waveform image, including

[0042] The heart sound two-dimensional waveform image and the electrocardiogram two-dimensional waveform image are concatenated in the vertical axis direction to obtain a two-dimensional heart sound electrocardiogram waveform image.

[0043] A heart sound electrocardiogram image neural network is used to extract a second feature vector, including

[0044] The local detail features of the two-dimensional heart sound and electrocardiogram waveform image are extracted using a deformable convolution module to obtain a local detail feature map; wherein, the deformable convolution module includes at least two sequentially connected first deformable convolution modules;

[0045] Based on the local detail feature map, global structural features and local detail features are fused using global deformable region of interest alignment to obtain a second feature vector.

[0046] A system for processing synchronous heart sound electrocardiogram signals, comprising,

[0047] The signal acquisition module is configured to synchronously acquire heart sound signals and electrocardiogram signals within a certain time period; wherein, the time period includes several cardiac cycles;

[0048] The signal segmentation module, located on the fog server, is configured to segment the heart sound signal according to length. The ECG signal is divided into several segments, resulting in a first heart sound signal; the ECG signal is then divided according to length. The signal is divided into several segments, resulting in a first electrocardiogram (ECG) signal; among them, the length is... It contains at least one cardiac cycle;

[0049] The preprocessing module, located on the fog server, is configured to preprocess the first heart sound signal segment and then normalize it to obtain the second heart sound signal segment; and to preprocess the first electrocardiogram signal segment and then normalize it to obtain the second electrocardiogram signal segment.

[0050] The first feature vector acquisition module is set on the cloud platform and is configured to vertically splice the second heart sound signal segment and the second electrocardiogram signal segment to obtain a one-dimensional heart sound electrocardiogram signal, and extract the first feature vector from the one-dimensional heart sound electrocardiogram signal using a heart sound electrocardiogram signal neural network.

[0051] The second feature vector acquisition module is set on the cloud platform and is configured to convert the second heart sound signal segment and the second electrocardiogram signal segment into a two-dimensional heart sound waveform image and an electrocardiogram waveform image, respectively; vertically stitch the two-dimensional heart sound waveform image and the two-dimensional electrocardiogram waveform image to obtain a two-dimensional heart sound and electrocardiogram waveform image; and extract the second feature vector using a heart sound and electrocardiogram image neural network.

[0052] The third feature vector acquisition module is configured to concatenate the first feature vector and the second feature vector to obtain the third feature vector.

[0053] The classification module is configured to perform classification using the Sigmoid function based on the third feature vector; and,

[0054] A judgment module is configured to count according to the classification result, and judge as an anomaly when the anomaly proportion exceeds a threshold value;

[0055] The one-dimensional heart sound electrocardio signal is extracted by a heart sound electrocardio signal neural network to obtain a first feature vector, including,

[0056] The one-dimensional heart sound electrocardio signal is extracted by a first convolutional layer to obtain a shallow feature map;

[0057] Based on the shallow feature map, a coordinate attention mechanism is used to extract time sequence features in the horizontal direction and correlation features in the vertical direction to obtain a correlation feature map;

[0058] Based on the correlation feature map, a second convolutional layer is used to extract deep features to obtain a deep feature map;

[0059] The one-dimensional heart sound electrocardio signal is connected to the deep feature map by jumping and the spatial feature dimension is reduced to obtain a spatial feature map;

[0060] The features of the spatial feature map are tiled and time features are extracted to obtain a first feature vector.

[0061] According to the foregoing analysis, a one-dimensional (1D) signal directly extracts time sequence characteristics but relatively weakly captures spatial features, and a two-dimensional (2D) image can effectively extract image spatial features but only time and frequency cannot reflect the most real information of the heart sound electrocardio signal, therefore, the present application tries to effectively fuse heart sound and electrocardio according to the characteristics of the electrical signal and the mechanical signal of the heart, uses an artificial intelligence model to extract one-dimensional (1D) signal features and two-dimensional (2D) image features, the two features are mutually referenced, verified, analyzed and identified to find more potential patterns in the heart sound electrocardio, improve the accuracy and robustness of heart failure diagnosis, and provide more accurate diagnosis information for heart failure clinic, which is more conducive to reducing the potential life danger of heart failure and other heart diseases.

[0062] After the above scheme is used, the present application uses artificial intelligence to jointly process the heart sound electrocardio signal, and the beneficial effects are reflected in:

[0063] (1) The present application more effectively captures global information, spatial information and time information, uses a heart sound electrocardio signal neural network to extract homologous and synchronous heart sound electrocardio time sequence characteristics but relatively weakly captures spatial features, and uses a heart sound electrocardio image neural network to extract homologous and synchronous heart sound electrocardio waveform image spatial features, thereby improving the accuracy and robustness of signal processing, and the obtained results can improve the accuracy of heart failure diagnosis;

[0064] (2) The present application directly faces homologous and synchronous heart sound electrocardio signals, unlike the traditional heart sound electrocardio joint process which has a manual mutual calibration process, and obtains more real features;

[0065] (3) The present invention utilizes global deformable region of interest alignment and adaptively adjusts the pooling window to achieve precise alignment of the target region, which is beneficial for extracting spatial features of homologous synchronous heart sound and electrocardiogram waveform images that have both stability and discriminativeness, and can be more adapted to dynamic monitoring of heart sound and electrocardiogram and diagnosis of heart failure in dynamic scenarios.

[0066] (4) This invention utilizes electrocardiogram to reflect changes in cardiac electrical activity, reflecting the chronotropic, conductive, and repolarotropic properties of the sinoatrial node, and heart sounds to reflect cardiac inertia. In particular, the mechanical activity disturbances of the heart caused by lesions of the cardiac conduction tissue occur before changes in electrocardiogram. Therefore, this invention learns the characteristics of synchronous electrocardiogram and heart sounds from the same source, which helps to quickly diagnose heart failure and overcomes the bias and limitations of only focusing on the single characteristics of heart sounds and electrocardiogram.

[0067] Compared with traditional diagnostic systems (or methods) for heart disease, this invention has the following advantages:

[0068] (1) Based on the synchronous heart sounds and electrocardiograms of the same source, the diagnosis of heart failure is more comprehensive and can effectively avoid misdiagnosis and missed diagnosis caused by relying solely on heart sounds or electrocardiograms, such as the complete disappearance of atrial electrical activity in atrial flutter and atrial fibrillation.

[0069] (2) The signal processing method of synchronous heart sound and electrocardiogram simultaneously learns the one-dimensional heart sound and electrocardiogram signal features and the two-dimensional heart sound and electrocardiogram waveform features. Compared with single-mode information, it is more comprehensive, realistic and rich, and has higher accuracy and stability.

[0070] (3) Give full play to the advantages of non-invasive data acquisition by heart sound sensor and electrocardiogram sensor, which can alleviate the shortage and uneven distribution of medical resources and facilitate daily home monitoring; by using the "cloud-edge-device" system structure deployment, it can support large-scale user data collection, which is conducive to early screening and follow-up of heart disease and overcomes the practical problems such as the lack of professional doctors and the uneven level of doctors. Attached Figure Description

[0071] Figure 1 This is a flowchart of the method of the present invention;

[0072] Figure 2 This is an architecture diagram of the system of the present invention;

[0073] Figure 3 These are waveforms of synchronous electrocardiograms and heart sounds obtained by this invention.

[0074] Wherein, (a) represents the heart sound signal and (b) represents the electrocardiogram signal. Detailed Implementation

[0075] The technical solution and beneficial effects of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] A preferred embodiment of the method for processing synchronous heart sound and electrocardiogram signals according to the present invention includes a training phase and an application phase. The training phase includes fog server-side preprocessing and cloud platform training.

[0077] The fog server-side preprocessing stage includes:

[0078] (A) Use the information acquisition module to connect to the edge server and synchronously record the same source of heart sounds and electrocardiogram signals;

[0079] (B) The sampling rate of both heart sound and electrocardiogram (ECG) signals is 2000Hz. A sliding window of size 4000 with a step size of 1000 is used to divide the signals into synchronous heart sound and ECG segments. The length of each heart sound and ECG signal segment is 2s. The heart sound signal segments are then normalized by Butterworth bandpass filter, wavelet transform denoising, and Z-score method. The ECG signal segments are then normalized by Butterworth bandpass filter, notch filter, and Z-score method.

[0080] The cloud platform training phase includes:

[0081] (a) Clean, homologous, synchronized heart sound and electrocardiogram signal segments were divided into training and test sets in a 4:1 ratio;

[0082] (b) Inputting one-dimensional synchronous heart sounds and ECG signal segments into a heart sound and ECG signal neural network to extract spatial and temporal features: longitudinally splicing the synchronous heart sounds and ECG signal segments yields the heart sound and ECG signal. The input is a convolutional layer with kernel size (1,5) and stride (1,2), which outputs a shallow feature map. The system employs a coordinate attention mechanism to extract temporal features of heart sounds and electrocardiograms in the horizontal direction and correlation features in the vertical direction, and then outputs a correlation feature map. Then input the data into a convolutional layer with kernel size (2,5) and stride (1,2) to extract deep feature maps. ; By using skip connections with deep features and an average pooling layer with a pooling window of (1,5) and a stride of (1,5), the spatial feature dimension is reduced, resulting in a spatial feature map. ; lay flat The features were analyzed using two bidirectional long short-term memory (BiLSTM) networks with 64 and 16 hidden nodes, respectively, to extract temporal features, ultimately yielding the ECG and heart sound signal feature vectors. ;

[0083] The method employs a coordinate attention mechanism to extract temporal features in the horizontal direction and correlation features in the vertical direction of electrocardiogram and heart sounds, and then outputs a correlation feature map, i.e., a shallow feature map. ∈R C×H×WOne-dimensional temporal features are obtained by performing average pooling in both the horizontal and vertical directions respectively. Association features Soon Processed separately as and H and W are the height and width of the input feature map, The number of channels; then the associated features obtained by flipping the vertical direction. and the time series features obtained in the horizontal direction By concatenating the features, we obtain the associated feature map. = , where [,] represents the splicing operation. for convolution, Represents a non-linear activation function;

[0084] The Bidirectional Long Short-Term Memory (BiLSTM) network is composed of a forward LSTM and a backward LSTM. ,in, and These represent the forward and backward LSTM respectively, and [,] indicates the concatenation operation. Represents input features;

[0085] (c) The synchronously generated heart sound and electrocardiogram (ECG) signal segments are converted into two-dimensional waveform images of heart sound and ECG, respectively, and input into a heart sound and ECG image network to extract image features: Specifically, the heart sound signal segments are first converted into two-dimensional waveform images of heart sound, and the ECG signal segments are converted into two-dimensional waveform images of ECG, and then... heart sounds, Two-dimensional ECG waveform images are stitched together along the vertical axis. Two-dimensional waveform images of heart sounds and electrocardiograms; then the... Two-dimensional waveform images of heart sounds and electrocardiograms as Original feature map A deformable convolutional module with kernel size (1, 3) and stride (1, 2) is input. After processing by the deformable convolutional module, a local detail feature map is output. The output local detail feature map is then used as input to the deformable convolutional module. This process is repeated three times to obtain local detail feature maps. , and Finally, the feature map output by the deformable convolution module is... The input global deformable region of interest (ROI) alignment module is used to fuse global structural features and local detail features to obtain the feature vector of the heart sound electrocardiogram image. ;

[0086] The deformable convolution module consists of deformable convolution, coordinate attention mechanism, and... Convolutional structures are used, where deformable convolutions concentrate sampling points to the region of interest using offsets. ,in This represents the offset of each point in the convolution kernel relative to the center point. This represents the weights at the corresponding positions of the convolution kernel. This indicates the feature map input to the deformable convolutional module. The element value at position , Indicates the output feature map The element value at that position;

[0087] The Globally Deformable Region of Interest Alignment (ROI Align) module divides the region of interest into... For each cell, four sampling points are selected at equal intervals, and the values ​​at the four locations are calculated using bilinear interpolation. In this context, point Q is one of the four coordinate positions within the cell, x and y are the corresponding values ​​of point Q on the x and y axes, points Q1, Q2, Q3, and Q4 represent the pixel points closest to point Q, and x1, x2, y1, and y2 represent the corresponding values ​​of these four pixel points on the x and y axes. The average value of the four positions is taken as the output.

[0088] The feature map Inputting ROI Align yields the feature vector of the heart sound and electrocardiogram image. ,Right now As the region of interest (ROI) input to the global deformable ROI alignment module, ROI Align: , obtain feature map and by Obtain its offset, add the offset to the grid cell position, and then perform ROI Align again: Obtain feature map vector ;

[0089] (e) Assemble the feature vector of ECG heart sound signal Heart sound and electrocardiogram feature vector Obtain the feature vector And use Sigmoid to map features to predicted probabilities. The system performs classification, with the results divided into two categories: normal and abnormal. 0 represents normal, and 1 represents abnormal.

[0090]

[0091]

[0092] The application phase includes the following steps:

[0093] Step S1: Use a signal acquisition module to acquire synchronous heart sounds and electrocardiogram signals from the same source, and ensure that the duration of each acquisition includes several cardiac cycles.

[0094] Step S2: The fog server receives the heart sound and electrocardiogram signals sent by the signal acquisition module, and processes them according to a certain length. Divide the heart sound and electrocardiogram signal into equal-length segments; this length Ensure that it contains at least one cardiac cycle; fill or discard any lengths less than L.

[0095] Step S3: For each equal-length heart sound and electrocardiogram signal segment mentioned in step S2, the fog server sequentially applies Butterworth bandpass filter, wavelet transform for noise reduction, and normalization to the heart sound signal segment, and sequentially applies Butterworth bandpass filter, notch filter, and normalization to the electrocardiogram signal segment.

[0096] Step S4: The cloud platform inputs the one-dimensional (1D) heart sound and electrocardiogram signal segments into the heart sound and electrocardiogram signal neural network to extract the heart sound and electrocardiogram signal feature vectors; and converts the heart sound and electrocardiogram signal segments into two-dimensional (2D) heart sound and electrocardiogram waveforms, inputs them into the heart sound and electrocardiogram image neural network to extract the heart sound and electrocardiogram image feature vectors.

[0097] Step S5: Concatenate the ECG and heart sound signal feature vectors and the heart sound and ECG image feature vectors, and use Sigmoid for classification.

[0098] The signal acquisition module includes a main control MCU unit, a heart sound sensor unit, an electrocardiogram (ECG) sensor unit, an analog-to-digital converter (ADC), a power supply module, and a wireless communication unit. The heart sound sensor unit uses ThinkLabs' THE ONE sensor, the ECG sensor unit uses Neurosky BMD 101, the main control MCU unit uses an STM32F103ZET6 chip, the ADC uses Analog Devices' AD7606, and the wireless communication unit uses a Realtek RTL8723BU supporting Bluetooth and Wi-Fi protocols.

[0099] The fog server is located at the edge network and utilizes an integrated Bluetooth and Wi-Fi gateway to receive and preprocess heart sound and ECG signals from the synchronous heart sound and ECG acquisition module, including bandpass filtering, notch filtering, and normalization. Specifically, the fog server employs a Cortex A8 processor chip, a 250GB solid-state drive, a Linux XenVMM operating system, an integrated gateway (Bluetooth and Wi-Fi), and a 10M / 100M network card, connecting to the cloud platform via the 10M / 100M network card.

[0100] The cloud platform employs virtualization technology to input received data into a neural network for heart sound and electrocardiogram (ECG) signals. Convolutional processing extracts shallow features, while a coordinate attention mechanism extracts temporal features in the horizontal direction and correlated features in the vertical direction, outputting a correlated feature map. Convolutional processing then extracts deep features, and average pooling reduces the spatial feature dimensionality. The spatial features are tiled, and a bidirectional long short-term memory network is used to extract ECG and heart sound signal feature vectors. The heart sound and ECG waveform image serves as input. The neural network for the heart sound and ECG image uses deformable convolution, a coordinate attention mechanism, and global deformable region of interest alignment (ROIAlign) to fuse local detail features and global structural features, thereby extracting ECG and heart sound image feature vectors that are both stable and discriminative. The classification module is responsible for concatenating the two output results and using the Sigmoid algorithm for classification.

[0101] Figure 2 The architecture of a system for processing synchronized heart sound and electrocardiogram signals is demonstrated, including a data acquisition terminal (comprising several signal acquisition modules), a fog server (for preprocessing), and a cloud platform. Figure 3 This display shows the heart sounds (PCG, Phonocardiogram), electrocardiogram (ECG), electrical activation time (EMAT), and left ventricular systolic time (LVST). Specifically, S1 (the first heart sound) is the sound produced when the mitral and tricuspid valves close at the start of ventricular systole; P point (the beginning of atrial systole); the QRS complex is the waveform generated by ventricular depolarization (electrical activity) on the ECG, representing the initiation of cardiac systole and is a key indicator for assessing heart rhythm, conduction function, and ventricular status; EMAT represents the time from the onset of the QRS complex to S1 (M1), reflecting the time required for the left ventricle to generate sufficient pressure to close the mitral valve, and is a parameter for evaluating left ventricular systolic function; EMAT% is the ratio of EMAT to RR, and an increase in this value indicates decreased left ventricular systolic function; LVST represents the left ventricular systolic time, i.e., from the first heart sound S1 to the second heart sound S2; LVST% is the ratio of LVST to RR.

[0102] The ECG and heart sound datasets of heart failure collected simultaneously in clinical settings were trained according to steps S1-S5. The training results were evaluated using sensitivity (TP / (FP+TN)), specificity (TN / (FP+TN)), and accuracy ((TP+TN) / (TP+FP+TN+FN)) as indicators, reaching 96.78%, 97.32%, and 96.62%, respectively. TP represents the number of true positives, TN represents the number of true negatives, FP represents the number of false positives, and FN represents the number of false negatives.

[0103] Furthermore, this invention supports the extraction of temporal features from one-dimensional homologous synchronous heart sound ECG signals and the extraction of spatial features from homologous synchronous heart sound ECG waveforms using a divide-and-conquer approach, which helps improve the accuracy of heart failure diagnosis. It also fully leverages the advantages of non-invasive data acquisition by heart sound and ECG sensors, facilitating daily home monitoring.

[0104] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0105] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0108] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0109] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for processing homochronous synchronous heart sound and electrocardio signals, characterized in that: The method comprises the following steps: synchronously acquiring heart sound signals and electrocardio signals in a time period, wherein the time period contains a plurality of cardiac cycles; The heart sound signal is divided according to length to obtain a plurality of first heart sound signal segments; the electrocardiogram signal is divided according to length to obtain a plurality of first electrocardiogram signal segments; wherein the length contains at least one cardiac cycle; preprocessing the first heart sound signal segment, then normalizing to obtain a second heart sound signal segment; preprocessing the first electrocardio signal segment, then normalizing to obtain a second electrocardio signal segment; vertically splicing the second heart sound signal segment and the second electrocardio signal segment to obtain a one-dimensional heart sound electrocardio signal, and extracting a first feature vector from the one-dimensional heart sound electrocardio signal by using a heart sound electrocardio signal neural network; converting the second heart sound signal segment and the second electrocardio signal segment into a heart sound two-dimensional waveform image and an electrocardio two-dimensional waveform image respectively, vertically splicing the heart sound two-dimensional waveform image and the electrocardio two-dimensional waveform image to obtain a two-dimensional heart sound electrocardio waveform image, and extracting a second feature vector from the two-dimensional heart sound electrocardio waveform image by using a heart sound electrocardio image neural network; splicing the first feature vector and the second feature vector to obtain a third feature vector; classifying by using Sigmoid according to the third feature vector; counting according to the classification result, and determining as abnormal when the abnormality ratio exceeds a threshold value; wherein the first feature vector is extracted from the one-dimensional heart sound electrocardio signal by using a heart sound electrocardio signal neural network, which comprises the following steps: extracting shallow features from the one-dimensional heart sound electrocardio signal by using a first convolutional layer to obtain a shallow feature map; extracting horizontal direction time sequence features and vertical direction correlation features from the shallow feature map by using a coordinate attention mechanism to obtain a correlation feature map; extracting deep features from the correlation feature map by using a second convolutional layer to obtain a deep feature map; jump connecting the one-dimensional heart sound electrocardio signal and the deep feature map and reducing the spatial feature dimension to obtain a spatial feature map; flattening the features of the spatial feature map and extracting time features to obtain the first feature vector.

2. The method of claim 1, wherein: dividing the heart sound signal according to length obtaining a plurality of first heart sound signal segments, comprising, said heart sound signal is divided by length the remaining part is padded or discarded if insufficient in length ​ dividing the electrocardiosignal according to length obtaining a plurality of first electrocardiosignal segments, comprising, The electrocardiosignal is divided by length The remaining part is padded or discarded if insufficient length .

3. The method of claim 1, wherein: The first heart sound signal segment is preprocessed, The method comprises the following steps: sequentially performing Butterworth band-pass filtering and wavelet transform denoising on the first heart sound signal segment; The first electrocardio signal segment is preprocessed, The method comprises the following steps: sequentially performing Butterworth band-pass filtering and notch filtering on the first electrocardio signal segment.

4. The method of claim 1, wherein: The method for extracting horizontal direction time sequence features and vertical direction correlation features from the shallow feature map by using a coordinate attention mechanism to obtain a correlation feature map comprises the following steps: averaging and pooling the shallow feature map in the horizontal direction and the vertical direction to obtain one-dimensional time sequence features and one-dimensional correlation features; splicing the one-dimensional correlation features after being flipped with the one-dimensional time sequence features to obtain the correlation feature map.

5. The method of claim 1, wherein: The method for vertically splicing the heart sound two-dimensional waveform image and the electrocardio two-dimensional waveform image to obtain a two-dimensional heart sound electrocardio waveform image comprises the following steps: The method for vertically splicing the heart sound two-dimensional waveform image and the electrocardio two-dimensional waveform image to obtain a two-dimensional heart sound electrocardio waveform image comprises the following steps: The method for extracting a second feature vector from the two-dimensional heart sound electrocardio waveform image by using a heart sound electrocardio image neural network comprises the following steps:

6. The method of claim 1, wherein: extracting local detail features from the two-dimensional heart sound electrocardio waveform image by using a deformable convolution module to obtain a local detail feature map; wherein the deformable convolution module contains at least two sequentially connected first deformable convolution modules; ​ Based on the local detail feature map, global structure feature and local detail feature fusion is realized by using global deformable region of interest alignment to obtain a second feature vector.

7. A processing system for homosynchronous heart sound and electrocardiogram signals, characterized by: Comprise, The signal acquisition module is configured to synchronously acquire heart sound signals and electrocardio signals within a time period; wherein the time period contains several cardiac cycles; The signal segment division module is arranged at the fog server end and is configured to divide the heart sound signal according to length to obtain a plurality of first heart sound signal segments; the electrocardiogram signal is divided according to length to obtain a plurality of first electrocardiogram signal segments; wherein the length contains at least one cardiac cycle. The preprocessing module is arranged at the fog server end and is configured to pre-process the first heart sound signal segment, and then normalize to obtain a second heart sound signal segment; pre-process the first electrocardio signal segment, and then normalize to obtain a second electrocardio signal segment; The first feature vector acquisition module arranged at the cloud platform end is configured to longitudinally splice the second heart sound signal segment and the second electrocardio signal segment to obtain a one-dimensional heart sound electrocardio signal, and extract a first feature vector from the one-dimensional heart sound electrocardio signal by using a heart sound electrocardio signal neural network; The second feature vector acquisition module arranged at the cloud platform end is configured to convert the second heart sound signal segment and the second electrocardio signal segment into a heart sound two-dimensional waveform image and an electrocardio two-dimensional waveform image respectively, longitudinally splice the heart sound two-dimensional waveform image and the electrocardio two-dimensional waveform image to obtain a two-dimensional heart sound electrocardio waveform image, and extract a second feature vector from the two-dimensional heart sound electrocardio waveform image by using a heart sound electrocardio image neural network; The third feature vector acquisition module is configured to splice the first feature vector and the second feature vector to obtain a third feature vector; The classification module is configured to classify by using Sigmoid according to the third feature vector; and The judgment module is configured to count according to the classification result, and judge as abnormal when the abnormality ratio exceeds a threshold value; The one-dimensional heart sound electrocardio signal is extracted by using a heart sound electrocardio signal neural network to obtain a first feature vector, which comprises, The one-dimensional heart sound electrocardio signal is extracted by using a first convolutional layer to obtain a shallow feature map; Based on the shallow feature map, a coordinate attention mechanism is used to extract horizontal direction time sequence features and vertical direction correlation features to obtain a correlation feature map; Based on the correlation feature map, a second convolutional layer is used to extract deep features to obtain a deep feature map; The one-dimensional heart sound electrocardio signal is connected with the deep feature map by jumping and the spatial feature dimension is reduced to obtain a spatial feature map; The features of the spatial feature map are tiled and time features are extracted to obtain a first feature vector.

Images