A method for constructing a multimodal feature fusion model for automatic classification of ECG rhythm

Abstract

The present invention relates to a method for constructing a multimodal feature fusion model for automatic classification of electrocardiogram (ECG) rhythms, belonging to the field of biomedical signal processing technology. The multimodal feature fusion model constructed in the present invention is a dual-channel deep neural network that utilizes a dual-channel bottleneck residual network combined with different attention mechanisms. It extracts multimodal features of different dimensions from one-dimensional ECG sequences and two-dimensional ECG time-frequency maps, performs feature fusion, and then utilizes the fused multimodal features to automatically classify arrhythmia signals. The algorithm of the present invention exhibits stable and reliable performance and is expected to have great potential for practical application.

Description

Technical Field

[0001] The present invention belongs to the technical field of biomedical signal processing and relates to a method for constructing a multimodal feature fusion model for automatic classification of electrocardiogram rhythms. Background Art

[0002] The electrocardiogram (ECG) is the most representative and important non-invasive tool for diagnosing cardiac abnormalities. The effectiveness of the standard 12-lead ECG in diagnosing cardiac abnormalities such as arrhythmias, myocardial infarction, or coronary artery occlusion has been demonstrated in numerous studies. Traditional ECG analysis methods rely primarily on manual ECG interpretation by experienced cardiovascular disease specialists. However, this method is labor-intensive and resource-intensive and has limited adoption, particularly in areas with limited medical access. Due to factors such as varying levels of expertise among physicians, individual differences between patients, and the diversity of ECG signals across individuals, inexperienced physicians often have low ECG accuracy, thus failing to meet the healthcare needs of the population. With advances in computer hardware technology and algorithms, numerous researchers worldwide have begun researching automatic ECG signal classification techniques based on computer-aided diagnosis systems (CADS). This technology can provide a reference for physicians and assist them in making ECG signal judgments. Therefore, to bridge the gap in expertise among physicians and improve diagnosis efficiency and accuracy, the application of automatic ECG signal classification techniques in clinical diagnosis is of great significance.

[0003] Over the past few decades, researchers have proposed a variety of CADS detection techniques. One type involves machine learning algorithms based on signal processing techniques and manually extracted expert features. Expert features extracted based on clinical knowledge and rule-based algorithms have good clinical interpretability and can achieve good accuracy in detecting certain ECG categories. However, these methods are difficult to generalize to other categories because some useful key information is lost during ECG segmentation and feature extraction, making it impossible for the model to learn all useful features. Another type involves end-to-end deep neural networks that do not require manual feature extraction. Researchers have developed a variety of deep learning networks, such as multilayer perceptrons (MLPs), convolutional neural networks (CNNs), recursive neural networks (RNNs), and long-short-term memories (LSTMs). These deep learning models have improved the accuracy of various classification tasks, achieving accuracy similar to that of clinical cardiologists.

[0004] At present, the research on the automatic diagnosis model of arrhythmia has the following three shortcomings:

[0005] (1) Due to the limitations of the dataset, most research on automatic diagnosis of arrhythmias is conducted on a single dataset, and the proposed methods and models do not have universality and generalization capabilities.

[0006] (2) Most of the research work on automatic diagnosis of arrhythmias is based on modeling and analysis of single-modal features extracted from one-dimensional ECG sequences. Based on the non-stationarity and complexity of ECG signals, extracting multi-modal features is conducive to optimizing the analysis results.

[0007] (3) Currently, many studies are based on 12-lead ECG signals. 12-lead ECG can provide richer arrhythmia information. Different leads reflect the electrical activity of the heart from different spatial angles, causing different leads to present different forms at the same time. However, how to effectively integrate the spatial feature forms of different leads is the focus of current algorithm research. Summary of the Invention

[0008] In view of this, the purpose of the present invention is to provide a method for constructing a multimodal feature fusion model for automatic arrhythmia classification, which uses a dual-channel deep neural network to extract features of different dimensions from one-dimensional ECG sequences and two-dimensional ECG time-frequency diagrams, respectively, and combines the attention mechanism to better allow the feature extraction network to focus on the key feature positions of the ECG wave band, thereby obtaining richer arrhythmia information and effectively realizing the automatic classification of various arrhythmia signals.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A method for constructing a multimodal feature fusion model for automatic classification of ECG rhythms is proposed. The model is a dual-channel deep neural network that combines a dual-channel bottleneck residual network with different attention mechanisms. Multimodal features of different dimensions are extracted from one-dimensional ECG sequences and two-dimensional ECG time-frequency maps, and then feature fusion is performed. The fused multimodal features are then used to achieve automatic classification of arrhythmia signals.

[0011] The method specifically comprises the following steps:

[0012] S1: De-noise the collected ECG signal to obtain a denoised one-dimensional ECG sequence;

[0013] S2: Perform short-time Fourier transform on the ECG signal processed in step S1 to obtain a two-dimensional ECG time-frequency map; since each ECG lead can be converted into a two-dimensional time-frequency spectrum image, in order to fully obtain the pathological feature information hidden in each lead, the present invention converts all 12 lead signals, and the resolution of each time-frequency map is uniformly set to 224×224. The input of the two-dimensional ECG time-frequency map is X2∈R 12×224×224 ;

[0014] S3: Construct and train a multimodal feature fusion model (EC-MAM) based on the attention mechanism. Specifically, by constructing a dual-channel bottleneck residual network, multimodal features of different dimensions are extracted from the one-dimensional ECG sequence and the two-dimensional ECG time-frequency graph respectively and feature fusion is performed. The dual-channel bottleneck residual network is combined with the channel-domain attention mechanism and the mixed-domain attention mechanism. The channel-domain attention mechanism is used to enhance the important features in the one-dimensional ECG sequence, and the channel-domain attention mechanism and the spatial-domain attention mechanism are used to enhance the important features in the two-dimensional ECG time-frequency graph. Redundant features are filtered out through multiple iterative training to further improve the accuracy of ECG signal recognition.

[0015] Furthermore, in step S1 , denoising the collected ECG signal specifically includes: using a notch filter to remove power frequency interference, and using an improved wavelet threshold algorithm to remove baseline drift and myoelectric interference.

[0016] The present invention uses an improved wavelet threshold algorithm to remove baseline drift and myoelectric interference with a wide frequency distribution range. Based on the shortcomings of traditional threshold setting, the present invention improves it. The improved threshold formula is as follows:

[0017]

[0018]

[0019] Among them, σ j is the standard deviation of the wavelet coefficients of each layer, λ j is the wavelet threshold of each layer, and N is the number of signal sampling points. (j,k) are the wavelet coefficients of each layer obtained after the noisy signal is decomposed by wavelet, where j represents the number of layers and k represents the number.

[0020] Traditional thresholds do not change with the number of decomposition layers. However, the noise distribution in ECG signals is random. After wavelet decomposition, the noise component in the signal decreases layer by layer as the number of decomposition layers increases, while the true ECG signal increases layer by layer. Therefore, using a fixed threshold at different decomposition layers will result in excessive removal of true signal coefficients or excessive residual noise components. The improved threshold can effectively retain the wavelet coefficients of the true signal while effectively removing the wavelet coefficients of the noise signal.

[0021] In view of the shortcomings of traditional soft and hard threshold functions, this paper proposes an improved threshold function, and the wavelet coefficients obtained from it are as follows:

[0022]

[0023] in, is the coefficient after wavelet threshold processing, w (j,k) is the coefficient before wavelet threshold processing. a is an adjustable parameter.

[0024] The above formula overcomes the shortcomings of soft and hard threshold functions. When the coefficient before processing approaches the threshold, the coefficient after processing is close to 0, ensuring continuity at the threshold point, thereby avoiding the oscillation phenomenon caused by the hard threshold function; when the coefficient before processing gradually increases, the coefficient after processing by the threshold function gradually approaches the coefficient before processing, effectively solving the constant deviation problem existing in the soft threshold function, so that the signal distortion is less after denoising.

[0025] Furthermore, in step S2, the time domain representation of the electrocardiogram is converted into a corresponding time-frequency spectrum image to reflect the frequency relationship that changes over time, which is then used for feature extraction of arrhythmia.

[0026] Furthermore, in step S3, the steps of constructing the EC-MAM model are as follows: the preprocessed one-dimensional ECG sequence and two-dimensional ECG time-frequency map are sent into the dual-channel bottleneck residual network structure in parallel, and important features enhanced by the channel attention mechanism and the hybrid attention mechanism are obtained, namely, sequence feature (Sequence Feature) and time-frequency feature (Time-FrequencyFeature); the sequence feature and time-frequency feature are processed by the feature fusion layer to obtain the fused feature (Fused Feature), and finally the classification result of the arrhythmia signal is obtained by processing through the classification layer.

[0027] Furthermore, in step S3, a dual-channel bottleneck residual network is constructed using the stacked one-dimensional bottleneck residual structure and the two-dimensional bottleneck residual structure.

[0028] Furthermore, in step S3, the one-dimensional bottleneck residual network structure after introducing the channel-domain attention mechanism includes: a one-dimensional convolutional layer, a one-dimensional batch normalization layer, an activation layer, a one-dimensional maximum pooling layer, a bottleneck residual module based on the channel-domain attention mechanism, a one-dimensional maximum pooling layer and a linear layer.

[0029] Furthermore, in step S3, the module structure of the channel domain attention mechanism (CA) is as follows: the input of the channel domain attention mechanism is a feature map F∈F that has been processed by several convolutions and pooling. L×C, where L represents the length of the feature map and C represents the number of feature channels; the feature map F is compressed by global average pooling to obtain the feature map F1; after the "compression" operation, an "excitation" module follows. The "excitation" module first uses a fully connected layer with a dimensionality reduction coefficient r, followed by a ReLU activation function, and then a dimension-increasing fully connected layer to restore the dimension to the number of feature channels C of the input; after the feature map F1 passes through the "excitation" module, a feature map F2 of size 1×C is obtained, and finally the feature map F2 is combined with the input feature map F∈F L×C Perform point multiplication to obtain the output feature map R∈R L×C .

[0030] Furthermore, in step S3, the two-dimensional bottleneck residual network structure after introducing the mixed-domain attention mechanism includes: a two-dimensional convolutional layer, a two-dimensional batch normalization layer, an activation layer, a two-dimensional maximum pooling layer, a bottleneck residual module based on the mixed-domain attention mechanism, a two-dimensional maximum pooling layer and a linear layer.

[0031] Furthermore, in step S3, the module structure of the hybrid-domain attention mechanism (HA) is to connect the spatial-domain attention module (SA) and the channel-domain attention module (CA) in a serial manner;

[0032] The input of the HA module is the ECG feature map F∈R that has been processed by several convolutions and pooling operations. w×h×c , w, h, c represent the width, height, and number of channels of the feature map respectively;

[0033] The channel-domain attention module (H-CA) in the hybrid-domain attention mechanism consists of two independent pooling operation branches: global maximum pooling, global average pooling, and then a fully connected layer with a dimensionality reduction coefficient r, followed by a ReLU activation function, and then a dimension-increasing fully connected layer to restore the dimension to the number of input feature channels; the channel attention weights in the last two branches are activated by the Sigmoid function and then multiplied with the input feature map F to obtain the feature map Then the feature map Add to the input feature map F to get the feature map T∈T w×h×c ;

[0034] The input of the spatial domain attention module (H-SA) in the hybrid domain attention mechanism module is the feature map T. The H-SA module consists of two independent parallel branches. Each branch performs different pooling operations on the feature map. Then, the spatial domain attention weight is activated by the Sigmoid function and multiplied with the input feature map T to obtain the feature map.

[0035] Finally, the output feature map of the H-SA module Added to the input feature map T of the H-SA module, the feature map R∈R w×h×c .

[0036] Furthermore, in step S3, the EC-MAM is trained, specifically: a cross entropy loss function is used as the loss function of the EC-MAM model and optimized, and finally a classification result of the arrhythmia signal is obtained.

[0037] The beneficial effects of the present invention are:

[0038] 1) The present invention extracts multimodal features. A dual-channel network is used to extract the one-dimensional time series features and two-dimensional time-frequency features of the twelve-lead ECG signal, and then a feature fusion layer is used to integrate the two modal features.

[0039] 2) The present invention uses a bottleneck residual structure to deepen the network layer and enhance the learning ability of deep neural networks. The bottleneck residual structure has the advantages of increasing network depth, strengthening feature transfer and reducing the number of parameters. In order to efficiently learn the features in the one-dimensional ECG sequence and the two-dimensional ECG time-frequency diagram, the present invention uses a stacked one-dimensional bottleneck residual structure and a two-dimensional bottleneck residual structure as a feature extraction network, and fuses the two types of extracted features through a feature fusion layer. Finally, the fused multimodal features are used as the classification basis to generate multi-label arrhythmia classification results through the classification layer.

[0040] The bottleneck residual structure is a special residual neural network structure that not only solves the problem that the performance of deep neural networks saturates or even degrades with increasing depth, but also has the advantages of increasing network depth, strengthening feature transfer and reducing the number of parameters. Therefore, the present invention uses a stacked bottleneck residual structure as a feature extraction network to facilitate the extraction of deeper hidden features.

[0041] 3) In order to further optimize the feature extraction network and enable the network to adaptively enhance the learning of important features and minimize the influence of secondary features, the present invention combines the attention mechanism with the dual-channel bottleneck residual feature extraction network. By utilizing the attention mechanism, the feature extraction network can learn the important features of different arrhythmia signals, give greater weights to those features that have greater discriminability for classification, and give smaller weights to features such as noise. Through multiple iterative training, redundant features are filtered out, and the recognition accuracy of ECG signals is further improved, so that the useful information in the one-dimensional ECG sequence and the two-dimensional ECG time-frequency diagram can be retained as completely as possible, reducing the loss of valuable information.

[0042] For one-dimensional ECG sequences, the introduction of a channel-domain attention mechanism allows the feature extraction network to pay more attention to the relationships between different channels. By aggregating the feature relationships between channels, it adaptively adjusts the feature values ​​between each channel. For two-dimensional ECG time-frequency maps, the introduction of a mixed-domain attention mechanism not only strengthens important channel features and suppresses useless channel features, but also efficiently learns areas of the two-dimensional time-frequency map that require special attention, thereby preserving the key features of the time-frequency map. The mixed-domain attention mechanism enables the two-dimensional bottleneck residual structure to simultaneously enhance important features in the spatial and channel domains. In the classification and processing of arrhythmia signals, the introduction of an attention mechanism can better enable the feature extraction network to focus on the key feature locations of the ECG band.

[0043] 4) The algorithm of the present invention has stable and reliable performance and is expected to have great potential for practical application.

[0044] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0046] Figure 1 This is a flow chart of the method for implementing arrhythmia classification and detection based on a deep neural network with feature fusion in the present invention;

[0047] Figure 2 is a segment of each arrhythmia signal in the embodiment of the present invention;

[0048] Figure 3 It is the preprocessing of input data of different lengths in the embodiment of the present invention;

[0049] Figure 4 is a comparison diagram of the ECG signal before and after filtering in an embodiment of the present invention;

[0050] Figure 5 1 is a time-frequency spectrum image corresponding to the I-lead electrocardiogram of four different arrhythmia diseases in an embodiment of the present invention;

[0051] Figure 6 is a time-frequency spectrum image converted using different window lengths in an embodiment of the present invention;

[0052] Figure 7 1 is a structural diagram of the EC-MAM model according to an embodiment of the present invention;

[0053] Figure 8 is a diagram of a residual block structure according to an embodiment of the present invention;

[0054] Figure 9 is a bottleneck residual structure diagram in an embodiment of the present invention;

[0055] Figure 10 This is a structural diagram of a CA module in an embodiment of the present invention;

[0056] Figure 11 This is a structural diagram of the HA module in an embodiment of the present invention;

[0057] Figure 12 is a structural diagram of the CA module in the mixed-domain attention mechanism in an embodiment of the present invention;

[0058] Figure 13 : This is a structural diagram of the SA module in the hybrid domain attention mechanism in an embodiment of the present invention;

[0059] Figure 14 It is a bottleneck residual structure based on different attention mechanisms in an embodiment of the present invention;

[0060] Figure 15 It is the confusion matrix of the model on the test set in the embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0062] See also Figures 1 to 15 To verify the effectiveness of the present invention, the present invention provides an implementation process for applying the present invention to a mixed data set to achieve automatic classification of ECG rhythms. The algorithm flow chart is as follows: Figure 1 As shown, the method includes the following steps:

[0063] Step S1: Obtaining ECG data.

[0064] The ECG data used in this embodiment can be ECG data directly collected from the subject, or data from a public database. Specifically, in order to illustrate the specific implementation effect of the present invention, the ECG data used in this embodiment comes from a data set in an existing public library. Including data from three public databases: CPSC 2018, CPSC 2018-Extra, and Georgia. These ECG signals are collected in a standard 12-lead format with a sampling rate of 500Hz, but the ECG records are of varying lengths, lasting between 6 and 60s. This embodiment uses these ECG signals to train, verify, and evaluate models to classify 9 typical arrhythmia signals. They are atrial fibrillation (AF), first-degree atrioventricular block (IAVB), left bundle branch block (LBBB), right bundle branch block (RBBB), premature atrial contraction (PAC), premature ventricular contractions (PVC), sinus bradycardia (SB), sinus tachycardia (STach), and normal (SNR). Figure 2 A snippet of each signal class is shown. To evaluate the performance of the proposed model, the dataset was randomly partitioned into 10 equally sized subsets: 8 subsets were used for model training, and the remaining two subsets were used for validation and testing, respectively. The dataset was randomly shuffled before partitioning to prevent hidden numerical patterns in the ECG data from influencing the training process, thereby preventing overfitting and thus preventing hidden patterns from influencing the ECG classification results.

[0065] Since the model constructed in this embodiment is dual-channel, the input of the model is a parallel dual input, including a one-dimensional ECG time series and a two-dimensional ECG time-frequency diagram. For the one-dimensional ECG time series, the following processing is performed according to the characteristics of the signal samples in the mixed data set. In the mixed data set, the length of the ECG signal samples ranges from 6 seconds to 60 seconds, with a median of 30 seconds. Since the bottleneck residual network receives signals with a constant length, the signals shorter than 30 seconds are padded with zeros, and the signals longer than 30 seconds are truncated to the last 30 seconds, as shown in FIG. Figure 3 As shown. The sampling frequency of the ECG signal itself is 500Hz, so the input of the one-dimensional ECG time series X1∈R 12×15000For the two-dimensional ECG time-frequency diagram, since each ECG lead can be converted into a two-dimensional time-frequency spectrum image, in order to fully obtain the pathological feature information hidden in each lead, this embodiment converts all 12 lead signals and uniformly sets the resolution of each time-frequency diagram to 224×224. Therefore, the input of the two-dimensional ECG time-frequency diagram is X2∈R 12×224×224 .

[0066] Step S2: Preprocess the data.

[0067] In order to improve the model performance and the accuracy of the classification results, the original data is preprocessed, including signal denoising, data normalization, and data enhancement.

[0068] The presence of noise in ECG signals will affect the quality of the signal, mask the morphological characteristics of the signal, and reduce the signal-to-noise ratio. Therefore, these interferences must be removed from the ECG signal to ensure accurate ECG analysis. There are three main sources of noise in ECG signals, namely baseline drift, power frequency interference, and myoelectric interference. The frequency range of baseline drift is between 0.05 and 1 Hz, the frequency range of myoelectric interference is between 5 and 2000 Hz, and the frequency of power frequency interference depends on the AC power frequency in different countries or regions, usually 50 Hz or 60 Hz. In this embodiment, based on the different frequency characteristics of power frequency interference, myoelectric interference and baseline drift, a notch filter is used to filter out the power frequency interference, and an improved wavelet threshold is used to eliminate baseline drift and myoelectric interference.

[0069] The wavelet threshold method is a method for denoising signals based on wavelet transform. Selecting an appropriate threshold and constructing an appropriate wavelet threshold function can achieve good denoising effects. This embodiment selects the db10 wavelet as the wavelet basis. Considering that the sampling frequency of the ECG sample is 500Hz, the ECG signal is decomposed into 8 scales using the wavelet transform. The frequency range of the 8th layer is 0-0.98Hz, which contains baseline drift noise. The high-frequency detail components of the first, second, and third layers mainly contain myoelectric interference. Therefore, the present invention uses an improved threshold and threshold function to perform threshold processing on the above four layers of wavelet coefficients, and then reconstructs the processed detail coefficients of each layer.

[0070] In the process of wavelet threshold denoising, the threshold plays an important role in the whole process as the boundary between noise and useful signal. The traditional threshold does not change with the number of decomposition layers, while the noise distribution in the ECG signal is random. After the ECG signal is decomposed by wavelet, the noise component decreases layer by layer as the number of decomposition layers increases, and the real ECG signal increases layer by layer. Therefore, the use of a fixed threshold in different decomposition layers will cause the coefficients of the real signal to be removed too much or too much noise component to remain, that is, there is a certain error between the overall estimate σ of the noise and the noise estimate of different decomposition layers. Based on the shortcomings of the traditional threshold setting, the present invention improves it. The improved threshold formula is shown in formula (1):

[0071]

[0072]

[0073] In formula (1) and formula (2), σ j is the standard deviation of the wavelet coefficients of each layer, λ j is the wavelet threshold of each layer, and N is the number of signal sampling points. (j,k) are the wavelet coefficients of each layer obtained after the noisy signal is decomposed by wavelet, where j represents the number of layers and k represents the number.

[0074] There are two types of traditional wavelet threshold functions, namely hard threshold function and soft threshold function. The processing principle of the hard threshold function is to directly set the signal within the threshold to zero, and retain the original signal for the signal outside the threshold. Although this method retains the original information outside the threshold and has good fidelity, it is discontinuous at the threshold. Therefore, after being processed by the hard threshold function, the signal will produce a large number of oscillation points. The processing principle of the soft threshold function is to also set the signal within the threshold to zero, but for the signal outside the threshold, when the signal is greater than 0, the original signal is subtracted from the threshold, and when the signal is less than 0, the original signal is added to the threshold. The soft threshold function solves the deficiency of the hard threshold function that is discontinuous at the threshold, but it will cause a constant deviation in the coefficients before and after threshold processing, thereby causing signal distortion. This embodiment proposes an improved threshold function to address the shortcomings of traditional soft and hard threshold functions. The wavelet coefficients thus obtained are as follows:

[0075]

[0076] In the above formula (3), w (j,k) is the coefficient before treatment, is the coefficient after wavelet threshold processing. a is an adjustable parameter. This embodiment determines the adjustment parameter a by combining the results of the denoising evaluation index (signal-to-noise ratio, mean square error). When a=100, the best denoising effect can be obtained. Figure 4As shown in FIG, after denoising, the noise interference in the ECG signal is reduced, and the characteristics of the ECG signal itself can be more clearly displayed, thereby making it easier to classify the ECG signal and improving the classification accuracy.

[0077] To mitigate the impact of noise and outliers, unify ECG features of different dimensions, and accelerate network training, this embodiment uses Z-score normalization. This method normalizes data based on the mean and standard deviation of the original data. The processed data has a mean of 0 and a variance of 1, conforming to a standard normal distribution and being dimensionless. As shown in the following formula, μ is the sample mean and σ is the data standard deviation. To ensure that the training and test data use the same data transformation, the same μ and σ are used to normalize the test and training sets.

[0078]

[0079] In order to further alleviate the overfitting phenomenon of the neural network, this embodiment uses data expansion to perform data enhancement, that is, scaling the ECG signal on the time axis to broaden and enrich the distribution of training data.

[0080] Step S3: Obtain a two-dimensional ECG time-frequency diagram.

[0081] Select different types of ECG records and use STFT to convert the time series signal of ECG lead I into the corresponding time-spectrum image. The visualization results of the time-spectrum image are as follows: Figure 5 As shown in the figure, since the frequency range of the ECG signal is approximately 0.05Hz to 100Hz, the signal energy is mainly concentrated in the range of 0.5Hz to 40Hz. In order to facilitate the observation of the detailed information of the time spectrum, the frequency range of the vertical axis of the figure is set to 0 to 50Hz. Figure 5 In the figure, the time-frequency spectrum images corresponding to lead I of four different ECG signals, NSR, RBBB, AF, and IAVB, are shown. It can be seen that there are large differences between the time-frequency spectrum images of different diseases, that is, different time-frequency domain feature information is displayed. Therefore, a deep neural network can be used to extract features and perform classification based on the two-dimensional ECG time-frequency spectrum, thereby achieving the purpose of automatic diagnosis of arrhythmias.

[0082] There are two key parameters that need to be determined when using the short-time Fourier transform, namely the window function and the window function length. This embodiment uses the Hanning window as the window function. The Hanning window is a smooth window function based on weighted cosine. The Hanning window is suitable for processing signals with multiple frequency components and complex spectral performance. Different characteristic waves in the ECG signal have different frequency characteristics. Therefore, the Hanning window is used to analyze the time-frequency diagram of the ECG signal. The specific formula is as follows:

[0083]

[0084] After selecting the analysis window function, it is necessary to determine the length of the window function. The window length has a great influence on the short-time Fourier transform. The time-frequency spectrum comparison images converted using different window lengths are as follows: Figure 6 When the window length is set too long, as shown in Figure 6 (a), the window length used is 512, which increases the amount of calculation and results in too low time resolution and insufficient time domain precision, making it impossible to effectively analyze the characteristics of the ECG signal in the time domain. Figure 6 (c) The window length used is 128, which results in too low frequency resolution, loss of ECG signal information, and inability to accurately analyze the frequency components of the ECG signal. Therefore, the optimal window length is finally selected as 256, as shown in Figure 6 As shown in (b), both frequency resolution and time domain resolution are taken into account, and the reasonable selection of the length of the window function is conducive to improving the analysis performance of the signal in the time and frequency domain.

[0085] Step S4: Construct a multimodal feature fusion model (EC-MAM) based on attention mechanism.

[0086] The multimodal feature fusion model (EC-MAM) based on the attention mechanism proposed in this embodiment is a two-channel deep neural network. Each channel extracts multimodal feature information of the one-dimensional ECG sequence and the two-dimensional ECG time-frequency map under the action of different attention mechanisms. The EC-MAM model structure diagram is shown in the figure. Figure 7 The basic process of the EC-MAM model is as follows: the preprocessed one-dimensional ECG sequence and two-dimensional ECG time-frequency map are fed into a dual-channel network structure in parallel, obtaining important features enhanced by the channel attention mechanism and hybrid attention mechanism, namely sequence features and time-frequency features. The sequence features and time-frequency features are processed in the feature fusion layer to obtain fused features, which are finally processed in the classification layer to obtain the arrhythmia signal classification results.

[0087] Furthermore, when constructing the model in this embodiment, the specific construction steps are as follows:

[0088] (1) Bottleneck residual module

[0089] In this embodiment, 1D CNN and 2D CNN are used to extract one-dimensional ECG sequence features and two-dimensional ECG time-frequency graph features respectively. In order to improve the learning ability of deep neural networks, in the EC-MAM model proposed in this embodiment, stacked convolutional pooling layers are used to extract the features of ECG signals. However, due to the existence of nonlinear activation functions, each input to output process is almost irreversible, which also causes a lot of irreversible information loss, that is, the deepening of the network easily causes the problem of gradient disappearance. Therefore, the residual network is used to solve the problem that the performance of deep neural networks is saturated or even degraded with increasing depth. The core of the residual network is the residual block, which is composed of a number of network layers, such as Figure 8 As shown, the residual block consists of a convolutional layer, a batch normalization layer, an activation layer, a dropout layer, a convolutional layer, a batch normalization layer, and an activation layer. After training, the network can determine which layers are redundant. After these redundant layers pass through the residual block, the input and output are identical. The residual block enables the network to achieve identity mapping, ensuring that the performance of deeper networks is at least comparable to that of shallower networks as the network deepens.

[0090] The bottleneck residual structure is a special residual structure, such as Figure 9 As shown in the figure, compared with the ordinary residual structure, this structure uses a 1×1 convolution kernel to first reduce and then increase the original number of channels. Reducing the number of channels first can make data training and feature extraction more effective and intuitive. The introduction of the 1×1 convolution kernel also greatly reduces the parameters of the network, reducing the amount of calculation while increasing the depth of the model.

[0091] (2) Attention Mechanism

[0092] The bottleneck residual structure has the advantages of increasing network depth, strengthening feature transfer and reducing the number of parameters. The convolution operation in the bottleneck residual structure is to fuse features in a local area, and fuse more features by increasing the receptive field, but the feature map generated by the convolution operation cannot utilize relevant information outside this area. In order to obtain global information, this embodiment introduces an attention module based on the bottleneck residual structure. First, the channel domain attention mechanism (CA) is introduced in the one-dimensional bottleneck residual structure. Taking the 12-lead one-dimensional ECG sequence as an example, the importance of the feature information contained in different channel directions is different. The channel domain attention mechanism focuses on the influence of the feature vector on each channel on the classification result. The channel domain attention module can learn the weight corresponding to each channel to indicate the importance of the channel to the result. The larger the weight, the more important the channel information. The structure diagram of the channel domain attention module used in this embodiment is as follows: Figure 10 As shown. The input of CA is the feature map F∈F that has been processed by several convolutions, pooling, etc. L×C. The feature map F is compressed by global average pooling to obtain the feature map F1. F1 can be called the channel feature description value. This step is also called "compression". In order to make full use of the aggregated information generated by the "compression" operation, an "excitation" operation is followed by the "compression" operation. The goal of this operation is to fully capture the channel dependency information. To achieve this goal, the function of the "excitation" operation needs to meet the following two conditions: first, it must be able to learn the nonlinear relationship between channels and enhance the nonlinear learning ability of the network; second, it must be able to learn non-mutually exclusive relationships to ensure that it can act on multiple channels. The gate mechanism of the activation function can meet these two conditions, so the two activation functions used are the ReLU function and the Sigmoid function. In addition, in order to control the complexity of the model and make it have better generalization ability, two fully connected layers are used to parameterize the gate mechanism. Therefore, the "excitation" module first uses a fully connected layer with a dimensionality reduction coefficient r, followed by a ReLU activation function, and then a dimension-increasing fully connected layer to restore the dimensionality to the number of input feature channels C. The dimensionality reduction coefficient r is a hyperparameter. In this embodiment, r=8 is selected. After the feature map F1 passes through the "excitation" module, the feature map F2 of size 1×C is obtained. F2 represents the importance of each channel and can also represent the correlation between channels. Finally, the feature map F2 is combined with the input feature map F∈F L×C Perform point multiplication to obtain the output feature map R∈R L×C , R is the feature map obtained after learning the channel domain attention.

[0093] Then, a hybrid domain attention mechanism (HA) is introduced into the two-dimensional bottleneck residual structure. The hybrid domain attention (HA) module connects the spatial domain attention module (SA) and the channel domain attention module (CA) in a serial manner. Taking the 12-lead two-dimensional ECG time-frequency graph as an example, not all areas on the time-frequency graph contain useful feature information. The spatial domain attention mechanism uses convolutional neural networks to efficiently learn areas that need special attention, and converts the spatial information in the original time-frequency graph into another space to preserve the key features in the time-frequency graph. The hybrid domain attention module makes up for the deficiencies of channel domain attention and spatial domain attention by adding attention to the spatial domain and channel domain of the two-dimensional time-frequency graph at the same time, so that the convolutional neural network can simultaneously enhance important features in the spatial domain and channel domain. The structure of the hybrid attention module HA designed in this embodiment is as follows: Figure 11 The input of the HA module is the ECG feature map F∈R that has been processed by several convolutions and pooling. w×h×c, w, h, c represent the width, height, and number of channels of the feature map, respectively. The channel-domain attention module (H-CA) in the hybrid-domain attention mechanism is used to enhance the features of important channels in the two-dimensional time-frequency map. Unlike the channel-domain attention module (CA) that enhances the one-dimensional ECG sequence, H-CA includes two independent pooling operation branches: global maximum pooling and global average pooling. The H-CA module structure is shown in the figure below. Figure 12 As shown. Since the channel attention weights in the H-CA module are activated by the Sigmoid function, each weight is limited to [0,1]. After the feature map is multiplied with the input feature map F, the signal will weaken after multiple iterations. Therefore, the identity mapping idea in the residual neural network is introduced to convert the output feature map Then add it to the input feature map F to get the feature map T∈T w×h×c , reducing the loss of important information. Feature map T∈T w×h×c The calculation formula is as follows:

[0094]

[0095] The feature map T calculated by the H-CA module captures the characteristic information of the two-dimensional ECG time-frequency map in the channel domain and assigns greater weight to important channels. The feature map is then serially input into the spatial domain attention module for spatial domain attention calculation to obtain the attention weight of the feature map in the spatial domain. The structure diagram of the spatial domain attention module (H-SA) in the hybrid domain attention mechanism is shown in the figure below. Figure 13 As shown. The input of the H-SA module is the feature map T∈T after the H-CA module. w×h×c , the H-SA module also includes two independent parallel branches, each branch performs different pooling operations on the feature map. The spatial domain attention weights obtained by the H-SA module are also activated by the Sigmoid function, so the output of the H-SA module is Added to the input feature map T of the H-SA module, the feature map R∈R w×h×c The calculation formula of R is as follows:

[0096]

[0097] (3) Bottleneck residual module based on attention mechanism

[0098] The model proposed in this embodiment uses a bottleneck residual structure based on different attention mechanisms to extract one-dimensional ECG sequence features and two-dimensional ECG time-frequency graph features. The specific network structure is as follows: Figure 14 shown.

[0099] Step S5: Train the constructed network model.

[0100] In this embodiment, for model training, in order to improve training efficiency, a small batch gradient descent algorithm is used. In each iteration, a small number of training samples are randomly selected to calculate the gradient and update the parameters. This can greatly reduce the number of iterations required for convergence and make the converged result closer to the effect of gradient descent.

[0101] The learning rate is a crucial hyperparameter when optimizing neural networks. In this example, the Adam optimization algorithm was used to train the model. Unlike traditional stochastic gradient descent, the Adam algorithm not only uses momentum as the parameter update direction but also adaptively adjusts the learning rate. In this example, the learning rate was initialized to 0.001, and after every 10 iterations, the learning rate was reduced to 50% of the original value.

[0102] The cross-entropy loss function was used as the model's loss metric. This function is well-suited for classification tasks and was therefore used during training. The dropout parameter in the model was set to 0.4. The experiment was trained for 50 epochs, with a batch size of 64.

[0103] Step S6: Classify the ECG data using the trained network model.

[0104] Evaluation indicators such as precision, recall, F1 score, receiver operating curve area (ROC), and accuracy are used to evaluate the effectiveness of the model. The higher the scores of these indicators, the better the classification effect of the model. As can be seen from Table 1, the model proposed in this embodiment has achieved good performance in the arrhythmia classification task based on 12-lead ECG. For 9 types of arrhythmias, the average F1 score reached 85.1%, the average accuracy reached 96.6%, and the average precision and average recall were both above 84%. When identifying arrhythmia signals of three categories, namely atrial fibrillation (AF), sinus bradycardia (SB), and normal ECG signals (SNR), the F1 scores were all above 0.9. Figure 15 The confusion matrix of the model proposed in this embodiment on the test set is shown, and the score on the diagonal of the confusion matrix is ​​the recall rate.

[0105] Table 1 Performance indicators of the EC-MAM model of the present invention on the test set

[0106]

[0107] Verification experiment:

[0108] In order to verify the effectiveness of the automatic arrhythmia diagnosis model proposed in this embodiment, two groups of ablation experiments were designed in this embodiment.

[0109] (1) The first group of ablation experiments

[0110] Model 1: One Dimensional Electrocardiogram Classification Network based on Single-model information and Attention Mechanism (1D-EC-SAM). This model extracts only one-dimensional ECG time series features. This model serves as a baseline to test the performance of single-modal features based on one-dimensional ECG time series on the arrhythmia classification task.

[0111] Model 2: Two Dimensional Electrocardiogram Classification Network based on Single-model information and Attention Mechanism (2D-EC-SAM). This model extracts only 2D ECG frequency-frequency features. This model serves as a baseline to test the performance of single-modal features based on 2D ECG frequency-frequency features on arrhythmia classification tasks.

[0112] Model 3: This model, proposed in this paper, is the Electrocardiogram Classification Network based on Multi-model information and Attention Mechanism (EC-MAM). This model integrates the one-dimensional time series features and two-dimensional time-frequency features of the twelve-lead ECG signal. This model was designed to verify whether multimodal feature fusion improves model performance.

[0113] In the first group of ablation experiments, the above three models were constructed. As shown in Table 2, the twelve-lead arrhythmia classification performance based on different modal features was compared. By comparing the F1 averages of the three models on nine disease classifications, it can be seen that EC-MAM>1D-EC-SAM>2D-EC-SAM. The single-modal model that only extracts one-dimensional ECG time series features has better classification effect on most disease types than the single-modal model that only extracts two-dimensional ECG time-frequency graph features. However, the multimodal feature model EC-MAM proposed in Example 1 has the best classification performance in the multi-label arrhythmia classification task. This verifies the effectiveness of fusing multimodal features to improve model performance.

[0114] Table 2 Performance comparison of the model proposed in the first set of ablation experiments and the proposed model on the test set

[0115]

[0116] (2) The second group of ablation experiments

[0117] Model 4: EC-MMI (Electrocardiogram Classification Network based on Multi-model Information): A multimodal ECG classification network. Compared to Model 3, this model removes the attention mechanism from the dual-channel network. This model was designed to verify the effectiveness of the attention mechanism in improving model performance.

[0118] Compared with the model EC-MAM proposed in the present invention, EC-MMI removes the attention mechanism in the dual-channel network. The comparison results are shown in Table 3. The model with the added attention mechanism has a significantly improved classification performance compared to the model without adding attention, and the F1 average value in the nine disease classifications has increased by two percentage points. It can be analyzed from Table 3 that the attention mechanism mainly improves the recognition accuracy of the two types of arrhythmia signals, PAC and PVC. Many models have difficulty in classifying these two types of arrhythmia signals. On the one hand, it may be that the number of PAC and PVC samples in the mixed data set is small, which makes it difficult for the model to classify these two types of signals; on the other hand, the pathological characteristics of PAC and PVC in the electrocardiogram are similar to those of other types of arrhythmia diseases, which also makes it easy for the model to make mistakes when classifying PAC and PVC. Therefore, the deep neural network with the added attention mechanism has a stronger feature extraction capability, can automatically identify more discriminative hidden features from the data, and make more accurate predictions for easily confused arrhythmia signals.

[0119] Table 3 Performance comparison of the proposed model and the proposed model on the test set in the second set of ablation experiments

[0120]

[0121]

[0122] To ensure a fair comparison, all model and training parameters were set identically. Compared to the other two models, the proposed EC-MAM achieved the best classification performance in the multi-label arrhythmia classification task. This demonstrates the effectiveness of integrating multimodal features in improving model performance.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for constructing a multimodal feature fusion model for automatic classification of electrocardiogram rhythms, characterized in that: The multimodal feature fusion model is a dual-channel deep neural network that uses a dual-channel bottleneck residual network combined with different attention mechanisms to extract multimodal features of different dimensions from one-dimensional ECG sequences and two-dimensional ECG time-frequency maps, perform feature fusion, and then use the fused multimodal features to automatically classify arrhythmia signals. The method specifically comprises the following steps: S1: De-noise the collected ECG signal to obtain a one-dimensional ECG sequence; S2: Performing short-time Fourier transform on the ECG signal processed in step S1 to obtain a two-dimensional ECG time-frequency map; S3: Construct and train a multimodal feature fusion model based on the attention mechanism, namely EC-MAM. Specifically, by constructing a dual-channel bottleneck residual network, multimodal features of different dimensions are extracted from the one-dimensional ECG sequence and the two-dimensional ECG time-frequency map, and feature fusion is performed. The dual-channel bottleneck residual network is combined with the channel-domain attention mechanism and the mixed-domain attention mechanism. The channel-domain attention mechanism is used to enhance the important features in the one-dimensional ECG sequence, and the channel-domain attention mechanism and the spatial-domain attention mechanism are used to enhance the important features in the two-dimensional ECG time-frequency map. Then, the discriminative features for the classification of arrhythmia signals are given a large weight, and the noise is given a small weight. Redundant features are filtered out through multiple iterative training. The steps for constructing the EC-MAM model are as follows: the preprocessed one-dimensional ECG sequence and two-dimensional ECG time-frequency map are fed into a dual-channel bottleneck residual network structure in parallel to obtain important features enhanced by the channel attention mechanism and hybrid attention mechanism, namely sequence features and time-frequency features; the sequence features and time-frequency features are processed by the feature fusion layer to obtain fused features, and finally the classification results of the arrhythmia signal are obtained by processing them by the classification layer; A dual-channel bottleneck residual network is constructed by stacking one-dimensional bottleneck residual structure and two-dimensional bottleneck residual structure; The one-dimensional bottleneck residual network structure after the introduction of the channel-domain attention mechanism includes: a one-dimensional convolutional layer, a one-dimensional batch normalization layer, an activation layer, a one-dimensional maximum pooling layer, a bottleneck residual module based on the channel-domain attention mechanism, a one-dimensional maximum pooling layer, and a linear layer.

2. The method for constructing a multimodal feature fusion model according to claim 1, wherein: In step S1, denoising the collected ECG signal specifically includes: using a notch filter to remove power frequency interference, and using an improved wavelet threshold algorithm to remove baseline drift and myoelectric interference; The calculation formula of the improved wavelet threshold algorithm is: in, is the coefficient after wavelet threshold processing, is the coefficient before wavelet threshold processing, Indicates the number of layers, Indicates the number; is an adjustable parameter, is the standard deviation of the wavelet coefficients of each layer, is the wavelet threshold of each layer, is the number of signal sampling points.

3. The method for constructing a multimodal feature fusion model according to claim 1, wherein: In step S3, the module structure of the channel domain attention mechanism is as follows: the input of the channel domain attention mechanism is a feature map that has undergone several convolution and pooling processes. ,in L represents the length of the feature map, C Indicates the number of feature channels; Feature Map The feature map is obtained by compressing the features through global average pooling ; After the "compression" operation, there is an "excitation" module, which first uses a dimensionality reduction coefficient A fully connected layer, followed by a ReLU activation function, followed by a dimension-increasing fully connected layer, so that the dimension is restored to the number of feature channels of the input ; Feature map After the "excitation" module, the size is Feature map , and finally the feature map With the input feature map Perform point multiplication to obtain the output feature map .

4. The method for constructing a multimodal feature fusion model according to claim 1, wherein: In step S3, the two-dimensional bottleneck residual network structure after the introduction of the mixed-domain attention mechanism includes: a two-dimensional convolutional layer, a two-dimensional batch normalization layer, an activation layer, a two-dimensional maximum pooling layer, a bottleneck residual module based on the mixed-domain attention mechanism, a two-dimensional maximum pooling layer and a linear layer.

5. The method for constructing a multimodal feature fusion model according to claim 4, wherein: In step S3, the module structure of the hybrid domain attention mechanism is to connect the spatial domain attention module and the channel domain attention module in a serial manner; The input of the mixed domain attention mechanism module is the ECG feature map after several convolution and pooling processes. , Represent the width, height, and number of channels of the feature map respectively; The channel domain attention module in the hybrid domain attention mechanism, namely H-CA, includes two independent pooling operation branches: global maximum pooling and global average pooling, and then adopts a dimensionality reduction coefficient A fully connected layer, followed by a ReLU activation function, followed by a dimension-increasing fully connected layer, so that the dimension is restored to the number of input feature channels; the channel attention weights in the last two branches are activated by the Sigmoid function and are compared with the input feature map. After point multiplication, we get the feature map , and then the feature map With the input feature map Add together to get the feature map ; The input of the spatial domain attention module in the hybrid domain attention mechanism module, namely H-SA, is the feature map The H-SA module consists of two independent parallel branches. Each branch performs different pooling operations on the feature map. Then, the spatial domain attention weights of the two branches are activated by the Sigmoid function and then compared with the input feature map. After point multiplication, we get the feature map ; Finally, the output feature map of the H-SA module Input feature map of the H-SA module T Add together to get the feature map R∈R w ×h×c .

6. The method for constructing a multimodal feature fusion model according to claim 1, wherein: In step S3, the EC-MAM is trained, specifically: a cross entropy loss function is used as the loss function of the EC-MAM model and optimized, and finally a classification result of the arrhythmia signal is obtained.

Images