Semi-supervised electrocardiogram myocardial infarction localization method based on hierarchical category constraints with different granularity

Abstract

The present invention discloses a semi-supervised electrocardiogram myocardial infarction localization method based on hierarchical different granularity category constraints. By building a self-learning deep learning neural network, the localization prediction of myocardial infarction is performed for a 12-lead electrocardiogram with only a small number of annotations. The network will output three levels of prediction results, and for unannotated images, the fine granularity, confusion granularity and coarse granularity are converted using hierarchical category conversion; due to the uneven distribution of data categories, the present invention uses cross-entropy-based category-aware dynamic loss constraints and hierarchical constraints to pay less attention to well-learned categories and take stronger attention to categories that are not well-learned. The method of the present invention can achieve better recognition effects with a small amount of annotations to a certain extent, thereby helping the intelligent medical system to analyze and diagnose experimental diseases in the future.

Description

Technical Field

[0001] The present invention belongs to the technical field of biomedical signal analysis and classification, and in particular relates to a semi-supervised electrocardiogram myocardial infarction localization method based on hierarchical different granularity category constraints. Background Art

[0002] Myocardial infarction is one of the five major killers threatening human health, with the characteristics of rapid onset and high mortality rate. Electrocardiogram, as the most widely used preliminary diagnostic evidence for myocardial infarction in clinical practice, plays an extremely important role. Due to the complexity of the heart and blood vessels, blockage of blood vessels at different locations may cause infarction at different locations of the heart. Therefore, how to accurately locate the infarction location through electrocardiogram has become a research focus. Since the location of myocardial infarction requires a lot of clinical experience, labeling electrocardiograms has become a process that not only requires a lot of time cost, but also requires experienced clinicians to make judgments on the location.

[0003] It is well known that deep learning networks require fine and accurate annotations as supervision information to make the model converge effectively. The larger the amount of annotations, the more likely the model will have a better training effect. Therefore, a more important question has come into the eyes of researchers, that is, how to use less annotation information while obtaining similar results as full annotation. At the same time, with the gradual development of deep learning, the focus of researchers' research has gradually shifted to studying fine-grained features, that is, distinguishing finer categories from coarser-grained categories, thereby improving the diagnosis and treatment of diseases. Among them, the location and classification of myocardial infarction is a common problem.

[0004] Since the heart is generally supplied by multiple blood vessels belonging to different levels, the blockage of these blood vessels will cause ischemia or necrosis of the different cavity walls affected by their blood supply. Generally speaking, according to the location of the heart to which these blood vessels belong, myocardial infarction is subclassified based on the anterior wall, outer wall, posterior wall, etc. (ALMI, AMI, ASMI, ILMI, IMI, IPLMI, IPMI, PMI, LMI). However, since the specific information of myocardial infarction often only exists in the subtle differences between the waveforms (Q waves, ST bands), this fine-grained classification task is difficult for the model. In previous work, the literature [Du N, Cao Q, Yu L, et al. FM-ECG: A fine-grained multi-label framework for ECG image classification [J]. Information Sciences, 2020, 549] used image-based electrocardiogram input to explore fine-grained categories. The model as a whole uses a transformer structure and integrates a spatial attention mechanism to extract features. The literature [Wang R, Fan J, Li Y. Deep multi-scale fusion neural network for multi-class arrhythmia detection [J]. IEEE journal of biomedical and health informatics, 2020, 24 (9): 2461-2472] uses two parallel network structures with different convolution kernels to perform convolution operations on the shallow features extracted from the backbone network, and then outputs the final result through the splicing of the two features and the attention mechanism. Although these methods have achieved good results, the model only extracts fine-grained features from the perspective of structural design, without considering the hierarchical relationship of the labels themselves, and thus ignores the information of mutual constraints between different levels. At the same time, since the features of these fine-grained categories have a high degree of similarity, how to distinguish between categories with high similarity that are easily confused with each other has become one of the focuses of academic research.

[0005] Fine-grained recognition has been an active field for many years. It can be achieved through powerful feature encoding, locating discriminative semantic parts or leveraging external information, but fine-grained annotation is usually time-consuming and requires experienced experts, because some subclasses of objects have marginal visual differences and are even difficult for humans to recognize. One way to reduce the cost of fine-grained labeling is to utilize weak labels, such as locating fine-grained parts through image-level labels; semi-supervised learning is another method, which usually utilizes a small set of labeled data and many unlabeled images for effective model training. Incorporating unlabeled data by adopting unsupervised consistency regularization is a popular method in current research.

[0006] In semi-supervised algorithms, to be specific, we have a large amount of unlabeled data and only a small amount of labeled data. In order to make better use of the existing data and give the model enough information, we need to use the unlabeled data as auxiliary information to improve the overall performance of the model. However, there are only a few works that aim to introduce semi-supervised learning into fine-grained image analysis. The reasons for this are mainly the following two challenges: ① In order to make better use of unlabeled data, how should we combine detailed information of different granularities? ② How should we deal with categories that are easily confused? Not only that, the scarcity of annotations in semi-supervised learning also exacerbates the above-mentioned dilemma. Summary of the invention

[0007] In view of the above, the present invention provides a semi-supervised electrocardiogram myocardial infarction localization method based on hierarchical category constraints of different granularity, which outputs prediction results of three different granularities through a deep learning neural network, and improves the recognition performance of the model through the combination of hierarchical fine-grained category constraints and category-aware dynamic loss functions.

[0008] A semi-supervised electrocardiogram myocardial infarction localization method based on hierarchical different granularity category constraints includes the following steps:

[0009] (1) Collect ECG signals from different patients with myocardial infarction category annotations, normalize these ECG signals to the same signal length, and divide them into training and test sets;

[0010] (2) A deep learning neural network is selected to predict the type of myocardial infarction. The prediction output is divided into three levels: fine-grained level, confusion level, and coarse-grained level.

[0011] (3) The ECG signals of the training set are input into the above-mentioned deep learning neural network in batches for training. A portion of the ECG signals in each batch uses category annotation information, while the remaining portion does not use category annotation information. Then, a loss function based on cross entropy loss + category-aware dynamic loss + cross-category self-constraint is designed for network iterative update;

[0012] (4) The ECG signals of the test set are input into the trained neural network to predict the corresponding myocardial infarction category results.

[0013] Furthermore, the deep learning neural network is a one-dimensional feature extraction network based on a ResNet structure and includes three feature output heads.

[0014] Furthermore, the prediction results output by the deep learning neural network are respectively P l3 ,P l2 ,P l1 , where P l3 Corresponding to the class probability prediction result at the fine-grained level, the size is C×D; P l2 Corresponding to the category probability prediction result of the confusion level, the size is B×D; P l1 It corresponds to the category probability prediction result of the coarse-grained level, with a size of 2×D, where D is the number of input signals in each batch, C and B are the number of categories at the fine-grained level and the confusion level, respectively, and C>B>2.

[0015] Furthermore, the specific process of training the deep learning neural network in step (3) is as follows:

[0016] 3.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate and optimizer;

[0017] 3.2 The ECG signals of the training set are input into the network in batches, and the network forward propagates the output to obtain the prediction results of three levels, and the loss function L between the prediction results and the labels is calculated. all ;

[0018] 3.3 According to the loss function L all The optimizer uses the gradient descent method to continuously iterate and update the model parameters until the loss function L all Convergence, training is completed; the loss function L all The expression is as follows:

[0019] L all =L su +L un

[0020] L su =L CAD +L ce +L cross

[0021] Where: L su is the loss function of the supervised part, L un is the loss function of the self-supervised part, L CAD is the category-aware dynamic loss function, Lce is the cross entropy function, L cross It is a three-layer category constraint function.

[0022] Furthermore, the cross entropy function L ce The expression is as follows:

[0023]

[0024] in: It means that the nth group of ECG signals with category labeling information in a batch is input into the network and the output is the category probability prediction result of level l. express The corresponding category annotation information, N is the number of ECG signals using category annotation information in a batch, l1, l2, and l3 represent the coarse-grained level, the confusion level, and the fine-grained level, respectively.

[0025] Furthermore, the category-aware dynamic loss function L CAD The expression is as follows:

[0026]

[0027] in: It means that the nth group of ECG signals with category labeling information in a batch is input into the network and the output is the category probability prediction result of level l. express The corresponding category annotation information, N is the number of ECG signals with category annotation information in a batch, l1, l2, l3 represent the coarse-grained level, the confusion level and the fine-grained level respectively, and w avg express The corresponding category weight.

[0028] Furthermore, for any category at level l, its category weight w avg The expression is as follows:

[0029]

[0030]

[0031]

[0032] Where: L avg represents the average loss function, L tar Represents the target loss function about the overall distribution of data, L m represents the cross entropy function of the category in the mth iteration, K m It represents the number of ECG signals belonging to this category in the mth iteration, and M represents the total number of iterations.

[0033] Furthermore, the three-layer category constraint function L cross The expression is as follows:

[0034]

[0035] in: It means that the nth group of ECG signals with category annotation information in a batch is input into the network and the category probability prediction result of the fine-grained level l3 is output. express The corresponding category labeling information, It means that the nth group of ECG signals with category annotation information in a batch is input into the network and the category probability prediction result about the confusion level l2 is output. express The corresponding category labeling information, It means that the nth group of ECG signals with category annotation information in a batch is input into the network and the category probability prediction result of the coarse-grained level l1 is output. express The corresponding category labeling information, N is the number of ECG signals using category labeling information in a batch.

[0036] Furthermore, the loss function L un The expression for is as follows:

[0037]

[0038] in: It means that the kth group of ECG signals in a batch without category labeling information is input into the network and the category probability prediction result about the fine-grained level l3 is output. It means that the kth group of ECG signals in a batch without category labeling information is input into the network and the category probability prediction result about the confusion level l2 is output. It means that the kth group of ECG signals in a batch without category labeling information is input into the network and the category probability prediction result of the coarse-grained level l1 is output. express Converted into the category probability label result of confusion level l2, express Transformed into the category probability label result of the coarse-grained level l1, express Converted into the category probability label result of the coarse-grained level l1, K is the number of ECG signals in a batch without category labeling information.

[0039] Furthermore, the category probability label result is a B-dimensional vector, B is the number of categories at the confusion level l2, and 1≤b≤B, for The value of the bth element in The maximum value of all class probability prediction values ​​belonging to the bth class in the confusion level l2; the class probability label result is a 2-dimensional vector, 2 is the number of categories at the coarse-grained level l1, and for The value of the ath element in The maximum value of all class probability prediction values ​​belonging to the a-th class of the coarse-grained level l1; the class probability label result is a 2-dimensional vector, for The value of the ath element in The maximum value of all class probability prediction values ​​belonging to the a-th class at the coarse-grained level l1.

[0040] The present invention uses a semi-supervised algorithm to solve the problem of fine-grained classification and recognition of electrocardiograms. The normalized multi-lead electrocardiogram signals are input into a deep learning neural network and the prediction results of three granularities are output. The network as a whole uses an end-to-end training method to form an overall closed loop, and uses three-level granularity constraints and category-sensitive loss functions to extract corresponding information for labeled and unlabeled data. The model of the present invention as a whole uses a self-learning and self-constraint method, so that it can have a better learning effect when the amount of annotations is insufficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the structure of the semi-supervised model of the present invention.

[0042] Figure 2 Schematic diagram of the model training method of the present invention.

[0043] Figure 3 Confusion matrix results (labels on the horizontal axis and predicted values ​​on the vertical axis) for semi-supervised training using 20% ​​training annotations, before adding the category-aware dynamic loss function and the three-level granularity category constraint.

[0044] Figure 4 Confusion matrix results (labels on the horizontal axis and predicted values ​​on the vertical axis) for semi-supervised training using 20% ​​training annotations after adding a category-aware dynamic loss function and a three-level granularity category constraint. DETAILED DESCRIPTION

[0045] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0046] The overall network model of the present invention is as follows Figure 1 As shown, the labeled ECG signal x a With the unlabeled ECG signal x u Input a deep learning neural network based on ResNet, and the model will output x for three levels of prediction P l3 , P l2 , P l1 For the supervised part, the cross entropy function, the category-aware dynamic loss function, and the three-level loss are used for simultaneous constraints; for the unsupervised part, the hierarchical category conversion expression is first used to perform prediction conversions at different levels, and then the three-level loss is used for constraints. The definition of the probability prediction result P is as follows:

[0047] P l3 ,P l2 ,P l1 =f(X)

[0048] Specifically:

[0049]

[0050] Where: C is the total number of categories.

[0051] For the supervised part, the cross entropy function, the category-aware dynamic loss function, and the three-layer category constraint are used for supervision, where the cross entropy function is expressed as:

[0052]

[0053] in: It means that the nth group of ECG signals with category labeling information in a batch is input into the network and the output is the category probability prediction result of level l. express The corresponding category annotation information, N is the number of ECG signals using category annotation information in a batch, l1, l2, and l3 represent the coarse-grained level, the confusion level, and the fine-grained level, respectively.

[0054] The category-aware dynamic loss function is:

[0055]

[0056]

[0057]

[0058]

[0059] The three-level category constraints are:

[0060]

[0061] Therefore, the overall loss function is:

[0062] L su =L CAD +L ce +L cross

[0063] For unlabeled data, hierarchical category conversion will be performed after the output of the model, and its expression is:

[0064] y l3-l2 =[max(p l3 ∈b)], 1≤b≤B

[0065] y l3-l1 =[max(p l3 ∈a)],a=1,2,

[0066] y l2-l1 =[max(p l2 ∈a)],a=1,2

[0067] Where: max(p l3 ∈b) is y l3-l2 The bth element value in is p l3 The maximum value of all class probability prediction values ​​belonging to the bth class of confusion level l2, max(p l3 ∈a) is y l3-l1 The value of the ath element in is p l3 The maximum value of all class probability prediction values ​​belonging to the a-th class of the coarse-grained level l1, max(p l2 ∈a) is y l2-l1 The value of the ath element in is p l2 The maximum value of all class probability prediction values ​​belonging to the a-th class at the coarse-grained level l1.

[0068] On this basis, three levels of constraints are used:

[0069]

[0070] Therefore, the supervision information L of the entire model all It is a combination of two parts:

[0071] L all =L su +L un

[0072] The process of the model training method of the present invention is as follows Figure 2As shown, we first use a random method to divide the labeled part of the dataset from the dataset, and the remaining part will be the unlabeled dataset, with the ratios being 0.2, 0.4, 0.6, and 0.8.

[0073] (x a ,y a ) = random_select(x,ratio)

[0074] Then we build a deep learning neural network ResNet. For network learning, the model as a whole uses the stochastic gradient descent algorithm to solve the three-part objective function. The iterative steps are as follows:

[0075]

[0076]

[0077] in: is the gradient operator, lr is the learning rate, w is the matrix parameter of the network, and C is the total number of classification categories.

[0078] The final condition for model convergence is as follows:

[0079] |L″-L′|≤ε

[0080] Where: L″ and L′ are the loss functions before and after iteration, ε is the convergence threshold, set to 10 -5 .

[0081] In order to verify the effectiveness of the present invention, we use 12-lead ECG signals of the heart for experiments. While using the method of the present invention to study the semi-supervised algorithm, in order to compare the experiments, we also use the same data set division to perform fully supervised learning on this part of the data. The data ratio is still 0.2, 0.4, 0.6, and 0.8; the overall input of the network is normalized so that all signal lengths are 1000 sampling points. The specific data settings are given in Table 1:

[0082] Table 1

[0083]

[0084]

[0085] For the training model, the deep learning algorithm network also uses the stochastic gradient descent algorithm for model convergence. Our experiment uses the NVIDIA 1080Ti GPU for accelerated computing. The overall training batch size is 4, the signal is normalized to a size of 12×1000, the model learning rate is 0.0025, and the entire training process requires 40 iterations; the fully supervised training data set is consistent with its corresponding semi-supervised labeled data, and uses the same generative segmentation network. Figure 3 Confusion matrix results for semi-supervised training using 20% ​​of the training annotations, before adding the category-aware dynamic loss function and the three-level granularity category constraint. Figure 4 The confusion matrix results after adding the category-aware dynamic loss function and three-level granularity category constraints for semi-supervised training using 20% ​​training annotations, where the horizontal axis is the label and the vertical axis is the predicted value, where light colors represent a smaller number and dark colors represent a larger number. As can be seen from the figure, before the category-aware dynamic loss function and three-level granularity category constraints were added, the prediction results in the same confusion set often had prediction errors (eg. the prediction misalignment problem of ALMI, AMI, and ASMI results), but after adding the two parts of supervision information we proposed, it can be seen from the figure that such problems are greatly reduced, the overall prediction results of the model are more accurate, and the predictions are more concentrated on the diagonal. This further illustrates from the side that the semi-supervised algorithm of the present invention can better handle the prediction error problem of confusion sets with similar features when the amount of annotations is small, so that it has better prediction results when using fewer annotations, thereby alleviating the problem of the traditional fully supervised algorithm requiring a large number of annotations to a certain extent, and better assisting the analysis of intelligent medical images when the amount of annotations is small.

[0086] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made to the present invention by those skilled in the art based on the disclosure of the present invention should be within the scope of protection of the present invention.

Claims

1. A semi-supervised electrocardiogram myocardial infarction localization method based on hierarchical different granularity category constraints, comprising the following steps: (1) Collect ECG signals from different patients with myocardial infarction category annotations, normalize these ECG signals to the same signal length, and divide them into training and test sets; (2) A deep learning neural network is selected to predict the type of myocardial infarction. The prediction output is divided into three levels: fine-grained level, confusion level, and coarse-grained level. (3) The ECG signals of the training set are input into the above-mentioned deep learning neural network in batches for training. A portion of the ECG signals in each batch uses category annotation information, while the remaining portion does not use category annotation information. Then, a loss function based on cross entropy loss + category-aware dynamic loss + cross-category self-constraint is designed for network iterative update; (4) The ECG signals of the test set are input into the trained neural network to predict the corresponding myocardial infarction category results.

2. The semi-supervised electrocardiogram myocardial infarction localization method according to claim 1, characterized in that: The deep learning neural network is a one-dimensional feature extraction network based on the ResNet structure and includes three feature output heads.

3. The semi-supervised electrocardiogram myocardial infarction localization method according to claim 1, characterized in that: The prediction results output by the deep learning neural network are P l3 , P l2 , P l1 , where P l3 Corresponding to the class probability prediction result at the fine-grained level, the size is C×D; P l2 The corresponding category probability prediction result of the confusion level is B×D; p l1 It corresponds to the category probability prediction result of the coarse-grained level, with a size of 2×D, where D is the number of input signals in each batch, C and B are the number of categories at the fine-grained level and the confusion level, respectively, and C>B>2.

4. The semi-supervised electrocardiogram myocardial infarction localization method according to claim 1, characterized in that: The specific process of training the deep learning neural network in step (3) is as follows: 3.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate and optimizer; 3.2 The ECG signals of the training set are input into the network in batches, and the network forward propagates the output to obtain the prediction results of three levels, and the loss function L between the prediction results and the labels is calculated. all ; 3.3 According to the loss function L all The optimizer uses the gradient descent method to continuously iterate and update the model parameters until the loss function L all Convergence, training is completed; the loss function L all The expression is as follows: L all =L su +L un L su =L CAD +L ce +L cross Where: L su is the loss function of the supervised part, L un is the loss function of the self-supervised part, L CAD is the category-aware dynamic loss function, L ce is the cross entropy function, L cross It is a three-layer category constraint function.

5. The semi-supervised electrocardiogram myocardial infarction localization method according to claim 4, characterized in that: The cross entropy function L ce The expression is as follows: in: It means that the nth group of ECG signals with category labeling information in a batch is input into the network and the output is the category probability prediction result of level l. express The corresponding category annotation information, N is the number of ECG signals using category annotation information in a batch, l1, l2, and l3 represent the coarse-grained level, the confusion level, and the fine-grained level, respectively.

6. The semi-supervised electrocardiogram myocardial infarction localization method according to claim 4, characterized in that: The class-aware dynamic loss function L CAD The expression is as follows: in: It means that the nth group of ECG signals with category labeling information in a batch is input into the network and the output is the category probability prediction result of level l. express The corresponding category annotation information, N is the number of ECG signals with category annotation information in a batch, l1, l2, l3 represent the coarse-grained level, the confusion level and the fine-grained level respectively, and w avg express The corresponding category weight.

7. The semi-supervised electrocardiogram myocardial infarction localization method according to claim 6, characterized in that: For any category at level l, its category weight w avg The expression is as follows: Where: L avg represents the average loss function, L tar Represents the target loss function about the overall distribution of data, L m represents the cross entropy function of the category in the mth iteration, K m It represents the number of ECG signals belonging to this category in the mth iteration, and M represents the total number of iterations.

8. The semi-supervised electrocardiogram myocardial infarction localization method according to claim 4, characterized in that: The three-layer category constraint function L cross The expression is as follows: in: It means that the nth group of ECG signals with category annotation information in a batch is input into the network and the category probability prediction result of the fine-grained level l3 is output. express The corresponding category labeling information, It means that the nth group of ECG signals with category annotation information in a batch is input into the network and the category probability prediction result about the confusion level l2 is output. express The corresponding category labeling information, It means that the nth group of ECG signals with category annotation information in a batch is input into the network and the category probability prediction result of the coarse-grained level l1 is output. express The corresponding category labeling information, N is the number of ECG signals using category labeling information in a batch.

9. The semi-supervised electrocardiogram myocardial infarction localization method according to claim 4, characterized in that: The loss function L un The expression is as follows: in: It means that the kth group of ECG signals in a batch without category labeling information is input into the network and the category probability prediction result about the fine-grained level l3 is output. It means that the kth group of ECG signals in a batch without category labeling information is input into the network and the category probability prediction result about the confusion level l2 is output. It means that the kth group of ECG signals in a batch without category labeling information is input into the network and the category probability prediction result of the coarse-grained level l1 is output. express Converted into the category probability label result of confusion level l2, express Transformed into the category probability label result of the coarse-grained level l1, express Converted into the category probability label result of the coarse-grained level l1, K is the number of ECG signals in a batch without category labeling information.

10. The semi-supervised electrocardiogram myocardial infarction localization method according to claim 9, characterized in that: The class probability label results is a B-dimensional vector, B is the number of categories at the confusion level l2, and for The value of the bth element in The maximum value of all class probability prediction values ​​belonging to the bth class of confusion level l2; the class probability label result is a 2-dimensional vector, 2 is the number of categories at the coarse-grained level l1, and for The value of the ath element in The maximum value of all class probability prediction values ​​belonging to the σth class of the coarse-grained level l1; the class probability label result is a 2-dimensional vector, for The value of the ath element in The maximum value of all class probability prediction values ​​belonging to the a-th class at the coarse-grained level l1.

Images