ECG classification method based on CvT-13 and multimodal image fusion

Abstract

The present invention relates to an ECG classification method based on CvT-13 and multimodal image fusion, which belongs to the field of signal processing technology. GADF, RP and GASF are used to convert ECG one-dimensional signals into three two-dimensional images of different modalities, and the three images are fused to generate a single image as the input of CvT-13. The multimodal fused image contains more features, and the CvT-13 model can take into account both local and global information, thereby effectively improving the classification performance.

Description

Technical Field

[0001] The invention belongs to the technical field of signal processing and relates to an electrocardiogram classification method based on CvT-13 and multimodal image fusion. Background Art

[0002] Electrocardiogram (ECG) records the electrical activity of the heart, thus providing a summary assessment of the heart's electrical activity. Electrocardiographs and ECG monitoring devices carry important cardiovascular information, but differences in recording environments, subject diseases, etc. make ECG signal classification a challenging task. The massive amount of ECG data highlights the importance of computer-assisted interpretation, and high-precision computer-assisted interpretation can save clinical experts a lot of time and energy.

[0003] In recent years, deep learning models have achieved good results in ECG classification tasks. Classification methods based on deep learning do not require professional knowledge about the data and selected features, and can adaptively learn representative features from the data, reducing excessive reliance on manual feature extraction. Convolutional Neural Networks (CNNs) can effectively extract features and have been widely used. They are mainly divided into two categories: 1D CNNs on time series and 2D CNNs on time-frequency spectrograms. The former takes the original ECG as input, while the latter focuses on the frequency characteristics of the ECG signal and uses the spectrogram as the input of two-dimensional CNNs for classification. However, both methods have single patterns and lack of time relationships, which have limitations. The literature "Yang Shuying, Gui Binbin, Chen Shengyong. Arrhythmia detection based on wavelet decomposition and 1D-GoogLeNet [J]. Journal of Electronics and Information Technology, 2021, 43(10): 3018-3027." proposes an accurate classification method based on wavelet decomposition and 1D-GoogLeNet. Wavelet decomposition obtains time-frequency domain information, and multidimensional data is used as input to improve classification accuracy. Reference "Peng Xiangdong, Pan Congcheng, Ke Zejun, et al. ECG classification method based on parallel architecture and spatiotemporal attention mechanism [J]. Journal of Zhejiang University (Engineering Edition), 2022, 56(10): 1912-1923." Use bidirectional long short-term memory network and convolutional neural network to extract features, capture the front and back dependencies of ECG sequence, and perform classification through parallel architecture and multi-way feature fusion spatiotemporal attention mechanism. Reference "Jun TJ, Nguyen HM, Kang D, et al. ECG arrhythmia classification using a 2-Dconvolutional neural network [J]. arXiv preprint arXiv:1804.06812, 2018." A deep two-dimensional CNN arrhythmia classification method based on grayscale ECG images is proposed. Converting one-dimensional ECG signals into two-dimensional images no longer requires noise filtering and feature extraction. The paper "Ullah A, Anwar SM, Bilal M, et al. Classification of arrhythmia by using deep learning with 2-D ECG spectral image representation[J]. Remote Sensing, 2020, 12(10): 1685." uses short-time Fourier transform to convert the one-dimensional ECG time series into a two-dimensional spectrogram. Experiments have shown that compared with 1D, 2D ECG provides more accurate heartbeat classification.The literature "Li Hongqiang, Wu Feifan, Cao Lu, et al. ECG signal classification algorithm based on improved deep residual network [J]. Journal of Tianjin Polytechnic University, 2022, 41(05): 65-72." Gramian angular fields (GAF) are used to convert one-dimensional ECG signals into two-dimensional images, and the improved deep residual network is used for classification. Compared with the limitations of single-modal signals, feature fusion collaborative classification achieves higher accuracy. The literature "Ahmad Z, Tabassum A, Guan L, et al. ECG heartbeat classification using multimodal fusion [J]. IEEE Access, 2021, 9: 100615-100626." proposed a multimodal image fusion (MIF) and multimodal feature fusion (MFF) framework. Compared with a single ECG modality, multimodal fusion achieved better performance. In the multimodal image fusion framework, a multimodal fusion image is generated as input after image fusion. Due to the different prominent features of various image conversion algorithms, the fused image carries more information, but the selected classification network fails to extract the fused features well, resulting in unsatisfactory classification results. In the multimodal feature fusion framework, features are extracted first and then fused to obtain unique and interdependent information, thereby achieving better classification performance. However, there are still some shortcomings. In this framework, a threshold fusion network is introduced. The feature extraction process of each module is independent of each other, and the weights determine the weights of each part when the feature is fused. Therefore, it is impossible to share specific data of different modalities, and there is a lack of global information association.

[0004] With the development of wearable monitors and mobile devices, deep learning classification methods based on manually customized features can no longer meet the needs, and automated solutions are urgently needed. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an ECG classification method based on CvT-13 and multimodal image fusion, which converts ECG one-dimensional signals into three two-dimensional images of different modalities through Gramian Summation Angular Field (GASF), Gramian Difference Angular Field (GADF) and Recurrence Plot (RP), and fuses them to generate a single image as the input of CvT-13. The multimodal fusion image contains more features, and the CvT-13 model can take into account both local and global information, thereby effectively improving the signal classification performance and having certain advantages.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] An electrocardiogram classification method based on CvT-13 and multimodal image fusion comprises the following steps:

[0008] S1: Divide the ECG dataset into training set and test set;

[0009] S2: For the ECG training set and test set, the Gram sum angular field GADF, recursive graph RP, and Gram difference angular field GASF image conversion algorithms are used to convert the one-dimensional ECG sequence into three types of two-dimensional images, forming three different modalities of the same ECG signal;

[0010] S3: The grayscale images of GADF, RP and GASF obtained after the ECG sequence conversion are used as three channels in the RGB image and combined into a color three-channel multimodal image;

[0011] S4: The converted three-channel multimodal image is resized to its original size as the input of the CvT-13 network;

[0012] S5: Use the training set to train the network, and use the test set to test it to get the classification results.

[0013] Furthermore, in step S1, it is determined whether equalization processing is required according to the situation of the ECG data set: if the difference in the number of different categories exceeds one order of magnitude, the SMOTE technology is used to balance the data of the small categories in the training set.

[0014] Further, the steps of the Grammar sum angular field GADF and Grammar difference angular field GASF image conversion algorithm in step S2 are as follows:

[0015] S21: For a given time series X = {x1, x2, ..., x n}, all values ​​are normalized to the interval [-1,1] or [0,1] by equation (1) (2):

[0016]

[0017] in Indicates that the time series in the rectangular coordinate system is normalized to the interval [-1,1], x i represents the i-th sample point in the sequence X, max(X) represents the maximum value in X, and min(X) represents the minimum value in X. Indicates that the time series in the rectangular coordinate system is normalized to the interval [0,1];

[0018] S22: Representing rescaled time series in polar coordinates Coordinates, encoding the value as the cosine of the angle and the timestamp as the radius, the formula is as follows:

[0019]

[0020] φ i Represents the angle cosine, converts the coordinates to polar coordinates, yes The i-th sample point, t i yes timestamp, N is a constant factor of the span of the canonical polar coordinate system;

[0021] S23: By considering the triangular sum or difference between each point, indicating the correlation between different time stamps, the definition formulas of GASF and GADF are as follows:

[0022]

[0023] Among them A GSF (i, j) represents the GASF obtained by the trigonometric transformation of the angle and A GDF (i, j) represents the GADF obtained by trigonometric transformation of the angle difference, φ i express The encoded angle cosine, φ j express The encoded angle cosine, φ n express The encoded cosine of the angle.

[0024] Further, the recursive graph RP algorithm in step S2 includes the following steps:

[0025] S24: Phase space reconstruction time series Y = {y1, y2, ..., y N}We get Y(n)={y(n),y(n+τ),...,y(n+(m-1)τ)}, n=1,2,...,N-(m-1)τ, where m is the embedding dimension and τ is the delay time;

[0026] S25: Calculate the distance r between Y(i) and Y(j) ij =||Y(i)-Y(j)||, where ||·|| represents the norm;

[0027] S26: According to R ij = Θ(λ-r ij ) calculates the recursive value, where λ represents the threshold and Θ(·) represents the Heaviside function, in the following form:

[0028] Furthermore, the CvT-13 network introduces convolution into two core parts of the ViT architecture:

[0029] The Transformer is divided into multiple stages to form a Transformer hierarchy. A Convolutional Token Embedding is included at the beginning of each stage to gradually reduce the sequence length while capturing local information.

[0030] Replace the Linear Projection before each self-attention block in the Transformer module with Convolutional Projection, using a depth-wise separable convolution operation;

[0031] The CvT-13 network includes 13 Transformer Blocks, integrating convolution into Transformer.

[0032] The beneficial effects of the present invention are as follows: the method of the present invention utilizes GADF, RP and GASF methods to convert ECG signals into three modal images respectively. During the conversion process, the time information is retained, and the information that cannot be displayed by one-dimensional signals can be highlighted from different angles. Multimodal image fusion regards the three modal images as three orthogonal channels of RGB images and fuses them into a color image. Therefore, the fused image has richer complementary information, thereby alleviating the weakness of a single modality in one-dimensional and two-dimensional forms, and it is easy to use the image classification network to perform subsequent classification tasks. The CvT-13 network can take into account both local and global features, has the advantages of convolutional neural networks and Transformers, can effectively extract the features of multimodal fusion images, accurately complete analysis and classification tasks, and effectively improve the classification performance of ECG signals.

[0033] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0035] Figure 1 This is a flow chart of the ECG classification method based on CvT-13 and multi-modal image fusion of the present invention;

[0036] Figure 2 This is the CvT network model architecture diagram. DETAILED DESCRIPTION

[0037] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0038] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0039] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0040] See also Figure 1 , Figure 2 At present, many studies focus on treating ECG signals as time series and applying one-dimensional signal classification methods. Compared with one-dimensional, the two-dimensional representation of ECG shows feature information from different angles. The present invention converts ECG signals into two-dimensional images and classifies them using image classification networks. Figure 1 As shown, the specific process is as follows:

[0041] 1) S1: According to the data set, determine whether equalization is needed. If the number of different categories differs by more than one order of magnitude, it is necessary to use SMOTE technology to balance the data of the small categories in the training set. The present invention uses the MIT-BIH data set and the PTB diagnostic data set for verification. Since there is data imbalance between different categories in the MIT-BIH data set, the small category training set is balanced, and the test set is not balanced to maintain the authenticity of the data. The PTB diagnostic data set does not require any balanced data processing and uses the original data set; the sizes of the MIT-BIH data set and the PTB diagnostic data set before and after equalization are shown in Tables 1 and 2.

[0042] Table 1

[0043]

[0044]

[0045] Table 2

[0046]

[0047] S2: The ECG training set and test set use GADF, RP and GASF image conversion algorithms respectively to convert the one-dimensional ECG sequence into three types of two-dimensional images, forming three different modes of the same ECG signal, and displaying data information in multiple aspects; the Gramian Angular Field (GAF) converts the ECG time series into a GAF image, and the ECG is mapped to the polar coordinate system, preserving the temporal characteristics of the ECG. According to the trigonometric functions used in the encoding, the Gramian Summation Angular Field (GASF) and the Gramian Difference Angular Field (GADF) can be obtained, which are defined as follows:

[0048] S201: For a given time series X = {x1, x2, ..., x n}, all values ​​are normalized to the interval [-1,1] or [0,1] through equations (1) and (2).

[0049]

[0050] S202: Representing rescaled time series in polar coordinates Coordinates, encoding the value as the cosine of the angle and the timestamp as the radius, the formula is as follows:

[0051]

[0052] in, yes The i-th sample point, t i yes timestamp, where N is a constant factor of the span of the canonical polar coordinate system.

[0053] S203: Indicate the correlation between different time stamps by considering the triangular sum / difference between each point. The definition formulas of GASF and GADF are as follows:

[0054]

[0055]

[0056] Recurrence Plots (RP) show important and easy-to-interpret information about the time scale, which helps in the diagnosis of dynamical systems and is widely used in the field of non-stationary signals. The RP algorithm is as follows:

[0057] S204: Phase space reconstruction time series Y = {y1, y2, ..., y N}We get Y(n)={y(n),y(n+τ),...,y(n+(m-1)τ)}, n=1,2,...,N-(m-1)τ, where m is the embedding dimension and τ is the delay time.

[0058] S205: Calculate the distance r between Y(i) and Y(j) ij =||Y(i)-Y(j)||, where ||·|| represents the norm.

[0059] S206: According to R ij = Θ(λ-r ij ) calculates the recursive value, where λ represents the threshold and Θ(·) represents the Heaviside function, in the following form:

[0060] S3: The grayscale images of GADF, RP and GASF obtained after the ECG sequence conversion are used as three channels in the RGB image, and combined to form a three-channel multimodal image, so the multimodal image is a color image. This multimodal image is not a direct conversion of the grayscale image into a color image, but is formed by the original data through different statistical methods. Therefore, the multimodal image synthesized by the present invention has the statistical characteristics of the ECG signal and carries more information than a single mode image;

[0061] S4: The converted 2D image is resized to 224×224 as the input of the CvT-13 network; the CvT-13 network introduces convolution into the two core parts of the ViT architecture. The overall architecture is shown in the figure below. Figure 2 As shown. First, the Transformer is divided into multiple stages to form a Transformer hierarchy. A Convolutional Token Embedding is included at the beginning of each stage to gradually reduce the sequence length while capturing local information. Second, the Linear Projection before each self-attention block in the Transformer module is replaced with a Convolutional Projection, and a deep separable convolution operation is used to enable the model to further capture local semantic information. CvT-13 has 13 Transformer Blocks, which integrates convolution into the Transformer, taking into account the advantages of local receptive field, weight sharing, dynamic attention, global receptive field, etc., and has achieved better performance in the field of image classification.

[0062] S5: Use the training set to train the network and obtain the classification results on the test set. In order to measure the classification effect of the designed method, the present invention uses accuracy Acc (accuracy), precision P (precision), recall R (recall) and F1 value as evaluation indicators, and the calculation formula is as follows:

[0063]

[0064]

[0065] Among them, TP = true positive, which indicates the number of samples correctly classified in the positive class; TN = true negative, which indicates the number of samples correctly classified in the negative class; FP = false positive, which indicates the number of samples incorrectly classified in the positive class; FN = false negative, which indicates the number of samples incorrectly classified in the negative class.

[0066] The method of the present invention utilizes multimodal image fusion to regard the three modal images as three orthogonal channels of RGB images, and fuse them into a color image. Therefore, the fused image has richer complementary information, thereby alleviating the weakness of a single modality in one-dimensional and two-dimensional forms, and can easily use the image classification network to perform subsequent classification tasks. The CvT-13 network can take into account both local and global features, has the advantages of convolutional neural networks and Transformers, can effectively extract the features of multimodal fusion images, and accurately complete analysis and classification tasks. Experiments have shown that the method of the present invention has achieved the best results in terms of accuracy, precision, recall rate and F1 value. The embodiment of the present invention does not require very strong signal prior knowledge, nor does it require too much expert input. It can automatically extract effective features from ECG signals for ECG type recognition, can improve recognition efficiency and accuracy, and can be applied to the recognition and classification of electrocardiograms by medical devices.

[0067] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above method embodiments. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An electrocardiogram classification method based on CvT-13 and multimodal image fusion, characterized in that: The following steps are involved: S1: Divide the ECG dataset into training set and test set; S2: For the ECG training set and test set, the Gram sum angular field GADF, recursive graph RP, and Gram difference angular field GASF image conversion algorithms are used to convert the one-dimensional ECG sequence into three types of two-dimensional images, forming three different modalities of the same ECG signal; S3: The grayscale images of GADF, RP and GASF obtained after the ECG sequence conversion are used as three channels in the RGB image and combined into a color three-channel multimodal image; S4: The converted three-channel multimodal image is resized to its original size as input to the CvT-13 network, which introduces convolution into two core parts of the ViT architecture: The Transformer is divided into multiple stages to form a Transformer hierarchy. A Convolutional Token Embedding is included at the beginning of each stage to gradually reduce the sequence length while capturing local information. Replace the Linear Projection before each self-attention block in the Transformer module with Convolutional Projection, using a depth-wise separable convolution operation; The CvT-13 network includes 13 Transformer Blocks, integrating convolution into Transformer; S5: Use the training set to train the network, and use the test set to test it to get the classification results.

2. The ECG classification method based on CvT-13 and multimodal image fusion according to claim 1, characterized in that: In step S1, according to the situation of the ECG data set, it is determined whether equalization processing is required: if the difference in the number of different categories exceeds one order of magnitude, the SMOTE technology is used to balance the data of the small categories in the training set.

3. The ECG classification method based on CvT-13 and multimodal image fusion according to claim 1, characterized in that: The steps of the Gram sum angular field GADF and Gram difference angular field GASF image conversion algorithm in step S2 are as follows: S21: For a given time series X = {x1, x2, ..., x n }, all values ​​are normalized to the interval [-1,1] or [0,1] by equation (1) (2): in Indicates that the time series in the rectangular coordinate system is normalized to the interval [-1,1], x i represents the i-th sample point in the sequence X, max(X) represents the maximum value in X, and min(X) represents the minimum value in X. Indicates that the time series in the rectangular coordinate system is normalized to the interval [0,1]; S22: Representing rescaled time series in polar coordinates Coordinates, encoding the value as the cosine of the angle and the timestamp as the radius, the formula is as follows: Among them, φ i Represents the angle cosine, converts the coordinates to polar coordinates, yes The i-th sample point, t i yes timestamp, N is a constant factor of the span of the canonical polar coordinate system; S23: By considering the triangular sum or difference between each point, indicating the correlation between different time stamps, the definition formulas of GASF and GADF are as follows: Among them A GSF (i, j) represents the GASF obtained by the trigonometric transformation of the angle and A GDF (i, j) represents the GADF obtained by the trigonometric transformation of the angle difference, φ i express The encoded angle cosine, φ j express The encoded angle cosine, φ n express The encoded cosine of the angle.

4. The ECG classification method based on CvT-13 and multimodal image fusion according to claim 1, characterized in that: The recursive graph RP algorithm described in step S2 includes the following steps: S24: Phase space reconstruction time series Y = {y1, y2, ..., y N }We get Y(n)={y(n),y(n+τ),...,y(n+(m-1)τ)}, n=1,2,...,N-(m-1)τ, where m is the embedding dimension and τ is the delay time; S25: Calculate the distance r between Y(i) and Y(j) ij =||Y(i)-Y(j)||, where ||·|| represents the norm; S26: According to R ij = Θ(λ-r ij ) calculates the recursive value, where λ represents the threshold and Θ(·) represents the Heaviside function, in the following form:

Images