Ventricular Activation Source Point Localization Method Based on Pretraining and Spatiotemporal Self-Attention Mechanism

Abstract

The present invention discloses a method for locating the origin point of ventricular activation based on pre-training and spatio-temporal self-attention mechanism. This method effectively improves and combines the self-attention mechanism and convolutional network in deep learning and introduces them into the localization of the origin point of ventricular tachycardia activation. Different from the traditional self-attention mechanism, the present invention simultaneously considers the self-attention features of electrocardiogram signals in both time and space dimensions, effectively fuses the features, uses convolution to extract the local features of the signals, and finally fuses the global spatio-temporal self-attention features and local feature information, and finally outputs the coordinates of the predicted activation origin point. In addition, the present invention also uses simulated data for pre-training to improve the localization accuracy of real experiments. The present invention can achieve a high localization accuracy without any preprocessing of the data.

Description

Technical Field

[0001] The present invention belongs to the technical field of electrophysiological analysis, and particularly relates to a method for locating the origin point of ventricular activation based on pre-training and spatio-temporal self-attention mechanism. Background Art

[0002] Every year, 17 million people worldwide die from cardiovascular diseases. Among them, patients with sudden cardiac death account for about 25% of all cardiovascular disease patients. Persistent ventricular tachycardia (VT) is extremely likely to cause sudden cardiac death. This kind of malignant arrhythmia generally needs to block the origin site of ventricular activation through catheter ablation surgery. However, this process is very time-consuming in clinical treatment and highly dependent on doctors' experience. Using a computer model to automatically locate the origin point of ventricular activation in 12-lead electrocardiogram will provide real-time guidance for doctors' clinical operations, and is expected to shorten the duration of the ablation process and improve the ablation efficacy. Therefore, how to quickly, accurately and non-invasively assist in locating the origin point of ventricular activation is an important issue in the combination of modern computers and medicine.

[0003] The QRS morphology on the electrocardiogram is largely determined by the coordinates of the origin point of ventricular activation. In clinical surgery, doctors usually need to invasively explore the origin point of ventricular activation point by point in the heart. Specifically, it is to physically stimulate different myocardial sites until the position where the abnormal QRS morphology (pace-mapping) appears again on the electrocardiogram is found. This trial-and-error method requires a large number of invasive catheter operations and a long operation time, and depends on the experience of clinicians, especially when VT has multiple origin points of ventricular activation. Therefore, using artificial intelligence technology to predict the origin location of ventricular tachycardia only using 12-lead electrocardiogram is a clinically significant topic.

[0004] Through research and retrieval, it is found that the literature [Missel R, Gyawali PK, Murkute JV, Li Z, Zhou S, AbdelWahab A, Davis J, Warren J, Sapp JL, Wang L. A hybrid machine learning approach to localizing the origin of ventricular tachycardia using 12-lead electrocardiograms. Comput Biol Med. 2020 Nov;126:104013] proposed and verified a hybrid model that combines population and patient-specific machine learning for rapid computer-guided rhythm mapping, achieving high accuracy on test data. However, the algorithm process is rather cumbersome and requires manual parameter adjustment, and only considers ventricular activation originating from the left ventricular endocardium. Some studies also consider transferring and adapting knowledge in simulated data to real data, such as the literature [M. Alawad and L. Wang, “Learning domain shift in simulated and clinical data: Localizing the origin of ventricular activation from 12-lead electro-cardiograms,” IEEE Trans. Med. Imag., vol. 38, no. 5, pp. 1172–1184] and the literature [S. Giffard-Roisin et al., “Transfer learning from simulations on a reference anatomy for ecgi in personalised cardiac resynchronization therapy,” IEEE Trans. Biomed. Eng., vol. 66, no. 2, pp. 343–353, Feb. 2019]; however, whether it is generating patient-specific simulated data or adapting simulated data to patient-specific anatomy, these methods require patient-specific anatomical data, which are difficult to obtain in patients undergoing electrocardiogram surgery. Summary of the Invention

[0005] In view of the above, the present invention provides a method for localizing the origin point of ventricular activation based on pre-training and spatio-temporal self-attention mechanism, which can automatically, quickly and accurately locate the specific position of the origin point of VT ventricular activation using 12-lead body surface potential data, represented by three-dimensional coordinates.

[0006] A method for locating the source point of ventricular activation based on pre-training and spatio-temporal self-attention mechanism, comprising the following steps:

[0007] (1) After importing the physiological data of real patients, use a cardiac modeling and simulation software tool to simulate the physiological characteristics of the heart at different abnormal points, including 12-lead body surface electrocardiogram data and the three-dimensional coordinates of the corresponding ventricular activation source points;

[0008] (2) Collect 12-lead body surface electrocardiogram data from different patients with ventricular tachycardia through a cardiac electrophysiological mapping tool, and record the three-dimensional coordinates of the corresponding ventricular activation source points of each group of 12-lead body surface electrocardiogram data in real time;

[0009] (3) Obtain a large number of samples through steps (1) and (2). Each group of samples includes 12-lead body surface electrocardiogram data and the three-dimensional coordinates of the corresponding ventricular activation source points. Among them, the samples obtained through step (1) are simulated simulation samples, and the samples obtained through step (2) are real experimental samples; then divide all samples into a training set and a test set;

[0010] (4) Build a network model based on pre-training and spatio-temporal self-attention mechanism, which is composed of an embedding layer, a position encoding layer, multiple cascaded global-local encoders, and a coordinate prediction layer connected in sequence from input to output;

[0011] (5) Use the 12-lead body surface electrocardiogram data in the training set samples as the model input, and the three-dimensional coordinates of the corresponding ventricular activation source points as the true value labels, so as to train the above network model to obtain a localization model for predicting the three-dimensional coordinates of the ventricular activation source points;

[0012] (6) Input the 12-lead body surface electrocardiogram data in the test set samples into the trained localization model, and the three-dimensional coordinates of the ventricular activation source points can be directly predicted and output.

[0013] Further, the specific implementation process of step (1) is as follows: First, import the patient case file constructed by the finite element method into the cardiac modeling and simulation software ECGSIM. Select the position of the simulated ventricular activation source point by clicking the mouse, and at the same time adjust the activation time, action potential amplitude, excitation duration, and resting potential value of the cardiac transmembrane potential of each node of the heart to simulate various arrhythmia conditions, and combine MATLAB software to obtain the simulated 12-lead body surface electrocardiogram data and the three-dimensional coordinates of the corresponding ventricular activation source points.

[0014] Further, the specific implementation process of step (2) is as follows: First, make the patient lie flat on the DSA (Digital Subtraction Angiography) operating table, collect the standard 12-lead surface electrocardiogram, and import the electrocardiogram into the CARTO3 system through the PIU jack; then move the positioning plate under the heart part to make the heart and the ablation catheter located in the mapped cardiac chamber be in the precise positioning area, and then use the CARTO3 system to select the appropriate position on the ventricular endocardium for three-dimensional electroanatomical mapping, and record the 12-lead surface electrocardiogram data and the three-dimensional coordinates of the corresponding ventricular activation source points to be ablated in real time.

[0015] Further, the embedding layer uses a linear layer with a ReLU activation function to map the input data into a multi-dimensional space.

[0016] Further, the position encoding layer is used to add a non-fixed position encoding to the input data that is automatically learned as the network is trained.

[0017] Further, the global-local encoder includes a global spatio-temporal attention calculation module, a local feature extraction module, and a data fusion layer. The data before and after the position encoding layer are respectively input into the local feature extraction module and the global spatio-temporal attention calculation module; the local feature extraction module consists of a one-dimensional convolutional layer and three one-dimensional dense dilated convolutional blocks D1 to D3 connected in sequence. The convolutional kernel size of the one-dimensional convolutional layer is 1×1, and the number of channels is set to be the same as the input; the convolutional kernel sizes of the three dense dilated convolutional blocks D1 to D3 are 7×7, 5×5, and 3×3 respectively, the dilation rate is set to 2 for all, and the padding mode is SAME, so that the output resolution size remains unchanged; the global spatio-temporal attention calculation module first calculates the self-attention matrix of the input data in space and time. The two self-attention matrices are concatenated through the concat function and then restored to the input dimension through a linear layer as the output of the spatio-temporal self-attention. Then, the output is subjected to residual connection and LayerNorm normalization and then input into the feed-forward network for feature extraction. The extracted features are subjected to residual connection and LayerNorm normalization again as the output of the global spatio-temporal attention calculation module; the data fusion layer weights and sums the outputs of the global spatio-temporal attention calculation module and the local feature extraction module, and then reduces the dimension to the required dimension through a linear layer.

[0018] Further, the coordinate prediction layer uses a linear layer with an output dimension of 3 to perform data prediction on the output of the last-level global-local encoder, so as to output the three-dimensional coordinates of the ventricular activation source point.

[0019] Further, the process of training the network model in step (5) is as follows: First, input the simulation samples in the training set into the model for training, calculate the loss function L between the prediction result of the model each time and the true value label, and continuously optimize the parameters in the network model by the backpropagation method with the goal of minimizing the loss function L until the loss function L converges, and then fix the model parameters as the initialization parameters for the next stage of training; then input the real experimental samples in the training set into the model for training, and repeat the above training process to obtain a localization model for predicting the three-dimensional coordinates of the ventricular activation source point.

[0020] The present invention effectively improves and combines the self-attention mechanism and convolutional network in deep learning and introduces them into the localization of the ventricular tachycardia activation source point. Different from the traditional self-attention mechanism, the present invention simultaneously considers the self-attention features of the electrocardiogram signal in the time and space dimensions respectively, effectively fuses the features, uses convolution to extract the local features of the signal, and finally fuses the global spatio-temporal self-attention features and local feature information, and finally outputs the coordinates of the predicted activation source point. In addition, the present invention also uses simulated data for pre-training to improve the localization accuracy of real experiments. The present invention can achieve a high localization accuracy without any preprocessing of the data.

[0021] The present invention proposes a new population-based deep learning model, which effectively combines the respective advantages of Transformer and DenseNet, and uses the global and local feature information extracted from the electrocardiogram signal to localize the origin of ventricular activation; as far as we know, no one has applied the adapted and improved Transformer to localize the origin of rapid ventricular activation. Therefore, the significance and creativity of the present invention are mainly reflected in the following points:

[0022] 1. The present invention proposes a deep learning model for non-invasive localization and segment classification of the origin of ventricular activation, effectively combines Transformer and DenseNet for the first time, and applies it to the electrocardiogram inverse problem.

[0023] 2. The algorithm of the present invention does not need to add complex artificial features or additional information, such as magnetic resonance imaging (MRI) or computed tomography (CT), and only needs a 12-lead electrocardiogram to effectively localize the origin of ventricular activation.

[0024] 3. When extracting features from the electrocardiogram signal, the present invention not only emphasizes the importance of global and local interaction information, but also needs to consider the time dimension and space dimension simultaneously when performing self-attention calculation.

[0025] 4. The method of the present invention can obtain a model with a relatively high accuracy without relying on simulated data or knowledge transfer. Brief Description of the Drawings

[0026] Figure 1 This is a schematic flow chart of the method for locating the ventricular activation source point of the present invention.

[0027] Figure 2 This is a schematic structural diagram of the network model constructed by the present invention.

[0028] Figure 3 This is a schematic structural diagram of the global-local encoder in the model of the present invention.

[0029] Figure 4 This is a schematic structural diagram of the spatio-temporal self-attention calculation module. Detailed implementation manners

[0030] To more clearly describe the present invention, the technical solutions of the present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0031] As Figure 1 shown, the method for locating the ventricular activation source point of the present invention based on pre-training and spatio-temporal self-attention mechanism includes the following steps:

[0032] S1. Import the physiological data of the case patient and obtain simulation data using modeling and simulation software.

[0033] Import the patient case file constructed by the finite element method into the cardiac modeling and simulation software ECGSIM. Select the position of the simulated ventricular activation source point by clicking the mouse, and at the same time adjust the activation time, action potential amplitude, excitation duration, and resting potential value of the transmembrane potential of each node of the heart to simulate various arrhythmia conditions. Combine with MATLAB software to obtain the simulated 12-lead body surface electrocardiogram data and the corresponding cardiac activation source point coordinates. A total of 6000 groups of simulation data are collected in this example, and the data format is mat file.

[0034] S2. Collect the 12-lead body surface electrocardiogram data of different patients with ventricular tachycardia through a cardiac electrophysiological mapping tool, and record the corresponding ventricular activation source points.

[0035] The patient lies flat on the DSA operation table, and the standard 12-lead body surface electrocardiogram is collected and connected to the PIU of the CARTO system; move the positioning plate under the heart to make the heart and the ablation catheter located in the mapped heart cavity be in the precise positioning area, and then use the CARTO3 system to select the appropriate endocardial position in the ventricle for three-dimensional electroanatomical mapping, and record the 12-lead body surface electrocardiogram data and the three-dimensional coordinates of the corresponding ventricular activation source points to be ablated in real time. A total of 14000 groups of simulation data are collected in this example, and the data format is mat file.

[0036] S3. Input the simulation data into the initialized network model for pre-training.

[0037] First, the dataset is divided into a training set, a validation set, and a test set according to a ratio of 8:1:1. The training set is used to train the network model parameters, the validation set is used to fine-tune the trained model to establish the final localization model, and the test set is used to test and verify the localization model.

[0038] During the pre-training process, the simulation data group is input into the randomly initialized network model for training to obtain a preliminary localization model. Specifically: First, the simulated 12-lead body surface electrocardiogram data is input into the network model one by one for training, and the loss function L between each prediction result of the network model and the true value label is calculated. The parameters in the network model are continuously optimized with the goal of minimizing the loss function L through the backpropagation method until the loss function L converges to obtain a preliminary localization model; in this example, the loss function L is defined as the mean square error, and the evaluation index is defined as the Euclidean distance D.

[0039] Aiming at the problem of insufficient localization accuracy in the existing technology, the present invention constructs a localization model with higher accuracy for the ventricular activation source point by designing a prediction framework that takes into account both local and global aspects and considering the correlation of the time and space characteristics of the electrocardiogram signal. As Figure 2 shown, the network model designed and constructed by the present invention includes: an embedding layer, a position encoding layer, 4 stacked global-local encoders, and a coordinate prediction layer. The global-local encoder includes a global spatio-temporal attention calculation module, a local feature extraction module, and a data fusion layer. As Figure 3 shown, the local feature extraction module includes three 1D dilated dense convolutional blocks and a 1×1 convolutional layer. The convolutional kernel sizes of the dilated dense convolutional blocks are 7×7, 5×5, and 3×3 respectively. The dilation rate of the dilated convolution is set to 2, and the padding mode is SAME, so that the output resolution size remains unchanged. The number of channels of the 1×1 convolutional layer is set to be the same as the input to ensure the same dimension during subsequent data fusion.

[0040] For the global spatio-temporal attention calculation module, based on the encoder of the traditional transformer, the position encoding is set to be learnable with the network. Since the QRS band of the electrocardiogram signal is basically in the middle of the whole waveform, the initialization is selected as a normal distribution between 0 and 1. After passing through the position encoding layer, the temporal self-attention and the spatial self-attention are calculated respectively. For the temporal self-attention: three new vectors Q, K, and V are obtained through a linear layer with an output dimension of 512 and split into 8 heads, and the global spatio-temporal attention calculation is performed on each head respectively. Specifically, as Figure 4As shown in the figure, first, the data of the last two dimensions of the K vector are swapped and then matrix-multiplied with the Q vector. The result obtained is divided by the square root of the dimension of the K vector to obtain the self-attention matrix. Then, the softmax function is applied to the last dimension to obtain the self-attention weights. Next, the self-attention scores are matrix-multiplied with the V vector to obtain the self-attention of one head. Finally, the self-attention of each head is concatenated together to obtain the final temporal self-attention; for the calculation of spatial self-attention, only the Q, K, and V in the temporal self-attention calculation are transposed and then the same operations are performed, and finally transposed back. The spatial self-attention and temporal attention are concatenated through the concat function and then restored to the dimension before input through a linear layer. The spatio-temporal self-attention is the output.

[0041] After the spatio-temporal self-attention calculation is completed, residual connection and LayerNorm normalization are performed, and then the input is fed into the feed-forward network for feature extraction. The feed-forward network consists of two linear layers; finally, a residual connection is added to obtain the output of the global spatio-temporal attention calculation module.

[0042] Since the importance of the global features and local features extracted for predicting the coordinates of the ventricular activation origin point is not the same, the data fusion layer multiplies the outputs of the global and local encoders by a learnable weight parameter respectively and then adds them arithmetically. The weight parameter is randomly initialized, and then the dimension is restored to the dimension before input to the global-local encoder through a linear layer.

[0043] Finally, the coordinate prediction layer is a linear layer with an output dimension of 3. Based on the output of the data fusion layer of the last global-local encoder, it maps the obtained features to the three-dimensional coordinate space and performs data prediction to obtain the three-dimensional coordinates of the predicted ventricular activation origin point.

[0044] S4. When the validation accuracy in the pre-training stage reaches the highest, the training is interrupted and all the parameters of the network are fixed and saved as a pth file.

[0045] S5. The real patient data obtained from the hospital is input into the model after fixing the pre-training parameters. The training set, loss function, and evaluation metrics are the same as those in the pre-training stage. At the same time, the parameter fixing is cancelled, so that all the parameters of the model can change with the training.

[0046] When the validation accuracy in the training stage based on real data reaches the highest, the training is interrupted and all the parameters of the network are fixed and saved as a pth file, and this model is used as the prediction of the three-dimensional coordinates of the ventricular activation origin point.

[0047] S6. The test set of the real data is input into the localization model of the ventricular activation origin point to obtain the test results.

[0048] The evaluation index data of the prediction results of the present invention are shown in Table 1 and Table 2. Among them, Table 1 shows the comparison results between the method of the present invention and other advanced machine learning methods. It can be seen that the present invention has the highest positioning accuracy among all methods, and the effect is also improved after pre-training. Table 2 shows the ablation experiment results of the method of the present invention, proving that each module of the model of the present invention has a positive impact on the positioning accuracy.

[0049] Table 1

[0050]

[0051] Table 2

[0052]

[0053] The above description of the embodiments is to enable those of ordinary skill in the art to understand and apply the present invention. Obviously, those who are familiar with the technology in this field can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative efforts. Therefore, the present invention is not limited to the above embodiments, and all improvements and modifications made by those skilled in the art according to the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A method for locating the origin point of ventricular activation based on pre-training and spatio-temporal self-attention mechanism, comprising the following steps: (1) After importing the physiological data of real patients, use a cardiac modeling and simulation software tool to simulate the physiological characteristics of the heart at different abnormal points, including 12-lead body surface electrocardiogram data and the three-dimensional coordinates of the corresponding origin points of ventricular activation; (2) Collect 12-lead body surface electrocardiogram data from different patients with ventricular tachycardia through a cardiac electrophysiological mapping tool, and record the three-dimensional coordinates of the origin points of ventricular activation corresponding to each group of 12-lead body surface electrocardiogram data in real time; (3) Obtain a large number of samples through steps (1) and (2). Each group of samples includes 12-lead body surface electrocardiogram data and the three-dimensional coordinates of the corresponding origin points of ventricular activation. Among them, the samples obtained through step (1) are simulated simulation samples, and the samples obtained through step (2) are real experimental samples; then divide all samples into a training set and a test set; (4) Build a network model based on pre-training and spatio-temporal self-attention mechanism, which is composed of an embedding layer, a position encoding layer, multiple cascaded global-local encoders, and a coordinate prediction layer connected in sequence from input to output; The global-local encoder includes a global spatio-temporal attention calculation module, a local feature extraction module, and a data fusion layer. The data before and after the position encoding layer are respectively input into the local feature extraction module and the global spatio-temporal attention calculation module; the local feature extraction module is composed of a one-dimensional convolutional layer and three one-dimensional dense dilated convolutional blocks D1-D3 connected in sequence. The convolutional kernel size of the one-dimensional convolutional layer is 1×1, and the number of channels is set to be the same as the input; the convolutional kernel sizes of the three dense dilated convolutional blocks D1-D3 are 7×7, 5×5, and 3×3 respectively, the dilation rate is set to 2, and the padding mode is SAME, so that the output resolution size remains unchanged; the global spatio-temporal attention calculation module first calculates the self-attention matrix of the input data in space and time. After the two self-attention matrices are concatenated by the concat function and passed through a linear layer to restore to the input dimension, it is used as the output of the spatio-temporal self-attention. Then, the output is subjected to residual connection and LayerNorm normalization and then input into the feed-forward network for feature extraction. The extracted features are subjected to residual connection and LayerNorm normalization again and then used as the output of the global spatio-temporal attention calculation module; the data fusion layer weights and sums the outputs of the global spatio-temporal attention calculation module and the local feature extraction module, and then reduces the dimension to the required dimension through a linear layer; (5) Use the 12-lead body surface electrocardiogram data in the training set samples as the model input, and the three-dimensional coordinates of the corresponding origin points of ventricular activation as the true value label, so as to train the above network model to obtain a localization model for predicting the three-dimensional coordinates of the origin points of ventricular activation; (6) Input the 12-lead body surface electrocardiogram data in the test set samples into the trained localization model, and the three-dimensional coordinates of the origin points of ventricular activation can be directly predicted and output.

2. The ventricular activation source point positioning method according to claim 1, characterized in that: The specific implementation process of step (1) is as follows: First, import the patient case file constructed by the finite element method into the cardiac modeling and simulation software ECGSIM. Select the position of the simulated ventricular activation source point by clicking with the mouse. At the same time, adjust the activation time, action potential amplitude, excitation duration, and resting potential value of the transmembrane potential of each node of the heart to simulate various arrhythmia conditions. Combine with MATLAB software to obtain the simulated 12-lead body surface electrocardiogram data and the three-dimensional coordinates of the corresponding ventricular activation source point.

3. The ventricular activation source point localization method according to claim 1, wherein: The specific implementation process of step (2) is as follows: First, let the patient lie flat on the DSA operating table, collect the standard 12-lead body surface electrocardiogram, and import the electrocardiogram into the CARTO3 system through the PIU jack; then move the positioning plate under the heart part so that the heart and the ablation catheter located in the mapped heart cavity are both in the precise positioning area. Then use the CARTO3 system to select the appropriate endocardial position in the ventricle for three-dimensional electroanatomical mapping, and record the 12-lead body surface electrocardiogram data and the three-dimensional coordinates of the corresponding ventricular activation source point to be ablated in real time.

4. The ventricular activation source point localization method according to claim 1, characterized in that: The embedding layer uses a linear layer with a ReLU activation function to map the input data into a multi-dimensional space.

5. The ventricular activation source point localization method according to claim 1, characterized in that: The position encoding layer is used to add a non-fixed position encoding to the input data that is automatically learned as the network is trained.

6. The ventricular activation source point positioning method according to claim 1, wherein: The coordinate prediction layer uses a linear layer with an output dimension of 3 to perform data prediction on the output of the last-level global-local encoder, thereby outputting the three-dimensional coordinates of the ventricular activation source point.

7. The ventricular activation origin point localization method according to claim 1, characterized in that: The process of training the network model in step (5) is as follows: First, input the simulated simulation samples in the training set into the model for training, calculate the loss function L between each prediction result of the model and the true value label, and continuously optimize the parameters in the network model by the backpropagation method with the goal of minimizing the loss function L until the loss function L converges, and then fix the model parameters as the initialization parameters for the next stage of training; then input the real experimental samples in the training set into the model for training, and repeat the above training process to obtain a positioning model for predicting the three-dimensional coordinates of the ventricular activation source point.

Images