Automatic identification method and system for STEMI and NSTEMI based on multi-modal deep learning

Abstract

The application provides an automatic identification method and system for STEMI and NSTEMI based on multi-modal deep learning, and belongs to the technical field of medical artificial intelligence, and comprises the following steps: step 1: a hybrid neural network framework is built; the hybrid neural network framework is optimized based on a total loss function to obtain an optimized neural network; 12-lead ECG signals are input into an input layer of the optimized neural network for preprocessing, and abnormal features of STEMI and NSTEMI are obtained based on a multi-scale CNN feature extraction module, a Transformer time series modeling module and a multi-task output head in sequence; the abnormal features of STEMI and NSTEMI are subjected to sliding window processing, and alarm output is performed based on a hierarchical alarm strategy; meanwhile, a heat map is generated based on Grad-CAM to verify whether the model attention area is reasonable by the heat map. The ECG is used as the basis to improve the identification accuracy and efficiency of STEMI and NSTEMI.

Description

Technical Field

[0001] The present invention relates to the field of medical artificial intelligence technology, and in particular to a method for automatically identifying STEMI and NSTEMI based on multimodal deep learning. Background Art

[0002] The current diagnosis of STEMI (acute ST-segment elevation myocardial infarction) and NSTEMI (acute non-ST-segment elevation myocardial infarction) mainly relies on the physician's visual interpretation of the electrocardiogram (ECG). However, there are problems such as time-consuming manual interpretation, easy misdiagnosis of minor ST-segment changes, as high as 15%-20% difference in interpretation between clinicians (especially in cases of critical ST-segment deviation), and insufficient specificity of traditional algorithms, resulting in poor ECG recognition effect.

[0003] Therefore, the present invention proposes an automatic identification method for STEMI and NSTEMI based on multimodal deep learning. Summary of the Invention

[0004] The present invention provides a method and system for automatic identification of STEMI and NSTEMI based on multimodal deep learning, which are used to solve the technical problems raised above.

[0005] The present invention provides a method for automatically identifying STEMI and NSTEMI based on multimodal deep learning, comprising:

[0006] Step 1: Build a hybrid neural network architecture, where the hybrid neural network architecture consists of an input layer, a multi-scale CNN feature extraction module, a Transformer time series modeling module, and a multi-task output head;

[0007] Step 2: Optimizing the hybrid neural network architecture based on the total loss function to obtain an optimized neural network, wherein the ST segment samples of the total loss function correspond to STEMI labels and the non-ST segment samples correspond to NSTEMI labels;

[0008] Step 3: The 12-lead ECG signal is input into the input layer of the optimized neural network for preprocessing, and the abnormal features of STEMI and NSTEMI are obtained based on the multi-scale CNN feature extraction module, the Transformer time series modeling module, and the multi-task output head.

[0009] Step 4: Perform sliding window processing on the abnormal features of STEMI and NSTEMI, and output alarms based on the hierarchical alarm strategy. At the same time, generate a heat map based on Grad-CAM for doctors to verify whether the model's focus area is reasonable through the heat map.

[0010] Preferably, the preprocessing includes: lead-specific normalization processing, wavelet threshold-based denoising processing, and data enhancement processing.

[0011] Preferably, the data enhancement processing includes:

[0012] Obtain a set of historical lead ECG signals, and input each historical lead ECG signal into the data enhancement model in turn to obtain an enhanced variable value based on each valley point and peak point in each historical lead ECG signal;

[0013] Obtaining a first average value of the enhanced variable value based on the peak point and a second average value of the enhanced variable value based on the valley point of each historical lead ECG signal, and constructing a first average vector and a second average vector;

[0014] Performing binary regression analysis on the first average vector and the second average vector to generate regression variable values;

[0015] performing aggregation processing on the first average vector and the second average vector in sequence to obtain a first aggregate distribution of the first average vector and a second aggregate distribution of the second average vector;

[0016] Filtering aggregate reference variable values ​​based on the first aggregate distribution and the second aggregate distribution;

[0017] Perform a global search for signal segments outside the normal range on the denoised 12-lead ECG signal;

[0018] Calculate the abnormal coefficient of each signal segment to obtain the abnormal variance;

[0019] If the abnormal variance is less than the preset variance, it is determined that all signal segments are in a stable state, and signal amplitudes of all signal segments are adjusted based on the regression variable value;

[0020] Otherwise, all signal segments are determined to be in an unstable state, and the first segment of the normal range closest to each signal segment is extracted;

[0021] Randomly select a variable value from the regression variable value and the aggregate reference variable value to adjust the signal amplitude of the first segment;

[0022] An enhanced processed signal is obtained according to the amplitude adjustment result.

[0023] Preferably, screening the aggregate reference variable value based on the first aggregate distribution and the second aggregate distribution includes:

[0024] Determine the average value and the number of aggregations under each first aggregation in the first aggregation distribution, and select the average value A1 of the first aggregation corresponding to the maximum number of aggregations;

[0025] Determine the average value and the number of aggregations under each second aggregation in the second aggregation distribution, and select the average value A2 of the second aggregation corresponding to the maximum number of aggregations;

[0026] The average value A1 and the average value A2 are averaged to obtain an aggregate reference variable value.

[0027] Preferably, the total loss function is constructed, including:

[0028] Constructing the cross entropy loss function ;

[0029] Where N represents the number of samples, and the number of samples of ST segment and non-ST segment is equal; Indicates the sample label of the i-th non-ST segment; represents the sample label of the i-th ST segment; represents the predicted probability that the i-th non-ST segment belongs to acute non-ST-segment elevation myocardial infarction; represents the predicted probability that the i-th ST segment belongs to acute ST-segment elevation myocardial infarction; lg represents the sign of the logarithmic function; 、 They represent the number of historical missed diagnoses of non-ST segment and ST segment based on N, respectively; 、 represent the error coefficients respectively;

[0030] Constructing contrast-constrained loss function ;

[0031]

[0032] in, Indicates the minimum interval threshold; Represents the ST segment feature under the Zth feature dimension Non-ST segment features The cosine distance of Represents feature dimension; represents the set of contrast constraints based on D;

[0033] Constructing task-balanced loss function ;

[0034] in, Indicates the i-th ST segment The predicted entropy of Indicates the i-th non-ST segment The predicted entropy of Based on The binary crossover function of Based on The binary crossover function of 、 They represent the balance compensation coefficient respectively;

[0035] Constructing a total loss function based on the cross entropy loss function, the contrast constraint loss function, and the task balance loss function;

[0036]

[0037] in, 、 Represent the adjustment coefficients respectively.

[0038] Preferably, a sliding window process is performed on the abnormal features of STEMI and NSTEMI, including:

[0039] The abnormal features of STEMI and NSTEMI are split into abnormal dimensions, and combined with the control relationship with the time point, the abnormalities in each dimension are marked at the corresponding time points on the input 12-lead ECG signal to obtain the abnormal distribution in each dimension;

[0040] Construct an anomaly set for each time point to determine the number of anomaly dimensions and the degree of anomaly, and match the first label obtained from the quantity-degree-label comparison table to the corresponding time point;

[0041] Starting from the initial time point of the input 12-lead ECG signal, the label sequence under each window is obtained according to the sliding window method;

[0042] Analyze the abnormal continuity of the label sequence under each window, and randomly screen N1 groups of three consecutive sequences for abnormal continuity. Combined with the overall characteristics of each group of three consecutive sequences and the correlation between the local characteristics and the overall characteristics of each sequence, analyze the continuous change direction of the abnormal continuity under each group. The continuous change direction includes: consistent direction change, enhanced direction change, attenuated direction change, and chaotic direction change.

[0043] Count the occurrence frequency of each change direction, and select the change direction with the maximum frequency as the final change direction;

[0044] The window adjustment coefficient is determined depending on all abnormal continuities under the final change direction, and the sliding window is resized in combination with a preset step size, and sliding window processing is performed.

[0045] Preferably, after verifying whether the model focus area is reasonable through the heat map, the method further includes:

[0046] After verifying that the model's focus area is unreasonable through the heat map, all offsets of the ST segment of the 12-lead ECG signal are obtained;

[0047] Perform a global analysis on all offsets using a global offset analysis model to obtain the corresponding first and second variations;

[0048] Performing mixed analysis on all offsets using a mixed offset analysis model to obtain corresponding third and fourth variations;

[0049] Obtaining a feature loss function based on the first change amount, the second change amount, the third change amount, and the fourth change amount;

[0050] The optimized neural network is optimized again using the feature loss function.

[0051] The present invention provides an electrocardiogram automatic recognition system based on multimodal deep learning, comprising:

[0052] Constructing a building module for building a hybrid neural network architecture, wherein the hybrid neural network architecture consists of an input layer, a multi-scale CNN feature extraction module, a Transformer time series modeling module, and a multi-task output head;

[0053] a framework optimization module for optimizing the hybrid neural network framework based on a total loss function to obtain an optimized neural network, wherein ST segment samples of the total loss function correspond to STEMI labels and non-ST segment samples correspond to NSTEMI labels;

[0054] A feature acquisition module is used to input the 12-lead ECG signal into the input layer of the optimized neural network for preprocessing, and sequentially obtain abnormal features of STEMI and NSTEMI based on the multi-scale CNN feature extraction module, the Transformer time series modeling module, and the multi-task output head;

[0055] The alarm output module is used to perform sliding window processing on the abnormal features of STEMI and NSTEMI, and output alarms based on a hierarchical alarm strategy. At the same time, a heat map is generated based on Grad-CAM for doctors to verify whether the model's focus area is reasonable through the heat map.

[0056] Compared with the prior art, the present invention has the following advantages:

[0057] Through multi-dimensional electrophysiological feature analysis and dynamic ST segment spatiotemporal modeling, the AMI diagnostic paradigm has been revolutionized. Its core innovation lies in breaking through the limitations of traditional electrocardiograms that rely on manual experience and building a precise revascularization decision support system.

[0058] Based on dynamic ST segment evolution patterns, lead-specific feature recognition, and risk stratification models, it can accurately screen patients requiring emergency PCI, reduce unnecessary coronary angiography examinations by 28%, and directly optimize medical resource allocation.

[0059] At the clinical application level, AI-ECG demonstrates wide applicability.

[0060] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0061] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0063] Figure 1 Flowchart of a method for automatically identifying STEMI and NSTEMI based on multimodal deep learning in an embodiment of the present invention;

[0064] Figure 2 2 is a structural diagram of an automatic identification system for STEMI and NSTEMI based on multimodal deep learning in an embodiment of the present invention;

[0065] Figure 3 This is a structural diagram of the CNN-Transformer hybrid model in an embodiment of the present invention;

[0066] Figure 4 Schematic diagram of the correspondence between multi-scale convolution kernels and ST segment features in an embodiment of the present invention;

[0067] Figure 5 This is an example diagram of the visualization of lead attention weights in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0069] Since the deep learning models in existing research are only based on convolutional neural networks (CNN), they lack the ability to model the dynamic characteristics of the electrocardiogram and may not be able to capture the temporal changes of the electrocardiogram signal. Although existing studies have mentioned the explanatory analysis of the model, they have not provided a specific interpretable output method (such as a heat map), making it difficult for clinicians to verify the diagnostic basis of the model. Existing models are mainly used for risk stratification, but they do not clearly distinguish between STEMI and NSTEMI, and lack accurate identification of specific types of myocardial infarction. Therefore, the present invention provides an automatic identification method for STEMI and NSTEMI based on multimodal deep learning, such as Figure 1 As shown, including:

[0070] Step 1: Build a hybrid neural network architecture, where the hybrid neural network architecture consists of an input layer, a multi-scale CNN feature extraction module, a Transformer time series modeling module, and a multi-task output head;

[0071] Step 2: Optimizing the hybrid neural network architecture based on the total loss function to obtain an optimized neural network, wherein the ST segment samples of the total loss function correspond to STEMI labels and the non-ST segment samples correspond to NSTEMI labels;

[0072] Step 3: The 12-lead ECG signal is input into the input layer of the optimized neural network for preprocessing, and the abnormal features of STEMI and NSTEMI are obtained based on the multi-scale CNN feature extraction module, the Transformer time series modeling module, and the multi-task output head.

[0073] Step 4: Perform sliding window processing on the abnormal features of STEMI and NSTEMI, and output alarms based on the hierarchical alarm strategy. At the same time, generate a heat map based on Grad-CAM for doctors to verify whether the model's focus area is reasonable through the heat map.

[0074] Preferably, the preprocessing includes: lead-specific normalization processing, wavelet threshold-based denoising processing, and data enhancement processing.

[0075] In this embodiment, the method can quickly and accurately identify STEMI and NSTEMI, reduce human interpretation differences (the difference rate is reduced to 4.3%, p<0.001) through standardized ECG feature quantitative analysis, and provide dynamic risk stratification. In particular, it can quickly screen out high-risk patients who require urgent coronary revascularization (such as percutaneous coronary intervention, PCI), thereby significantly shortening the critical treatment time window and improving patient prognosis.

[0076] In this embodiment, the lead-specific normalization process is:

[0077] For input 12-lead ECG signal The matrix X has dimensions of 12×T and belongs to the real number domain R (T=5000 samples, 10 seconds, 500 Hz) for lead-by-lead normalization:

[0078]

[0079] in:

[0080] : Represents the value of variable i after standardization.

[0081] Xi: represents the value of the original variable i.

[0082] μi: represents the mean of variable i.

[0083] σi: represents the standard deviation of variable i.

[0084] i∈{I,II,…,V6}: Indicates that the formula applies to a specific set of variables (for example, the lead signals in an electrocardiogram). Because ECG signal amplitudes can vary significantly between leads (for example, the amplitude of chest lead V2 can be 10 times higher than that of limb lead aVR), each lead is independently normalized to prevent the model from being dominated by high-amplitude leads. This ensures that all leads are of the same scale (mean 0, standard deviation 1), eliminating amplitude differences and facilitating kernel weight learning.

[0085] The denoising process based on wavelet threshold is:

[0086] ECG noise primarily comes from myoelectric interference (high frequency), baseline drift (low frequency), and power-frequency noise (50 / 60 Hz). We selected a Symlet wavelet with a waveform similar to the QRS wave (symmetry and tight support) that better preserves the ST segment morphology. We used the Symlet-5 wavelet to decompose the ECG signal and soft-thresholded the high-frequency coefficients:

[0087] The threshold λ is adaptively adjusted based on the noise standard deviation σ to retain the low-frequency components of the ST segment (0.5-5Hz)

[0088]

[0089] in:

[0090] DWT(X): Performs discrete wavelet transform on matrix X.

[0091] Threshold(·,λ): performs threshold processing on the result of DWT, where λ is the threshold parameter.

[0092] IDWT(·): Perform an inverse discrete wavelet transform (Inverse Discrete Wavelet Transform) on the result after threshold processing to obtain the final matrix X^.

[0093] Among them, the threshold (Donoho threshold), σ is the noise standard deviation, λ is adaptively adjusted based on the noise standard deviation σ, retaining the low-frequency component of the ST segment (0.5-5Hz)

[0094] For data enhancement processing, common methods are as follows:

[0095] TimeWarping: simulates real-world heart rate variations (60-100 bpm) and randomly selects a time scaling factor , linear interpolation adjusts the signal length.

[0096] Where: α: represents a random variable. U(0.9,1.1): represents a uniform distribution (UniformDistribution), with parameters in the interval [0.9,1.1]. Lead mask: randomly discards one lead to simulate electrode detachment, improving the model's robustness to poor electrode contact.

[0097]

[0098] in:

[0099] Xmasked: represents the matrix after mask operation.

[0100] X⊙M: represents the element-wise product (Hadamard product) of matrix X and mask matrix M.

[0101] : The mask matrix M is a 12-dimensional binary vector, each element of which can only be 0 or 1.

[0102] ∑M=11: There are 11 elements of 1 and 1 element of 0 in the mask matrix M.

[0103] In this embodiment, for the construction of hybrid neural network, such as Figure 3 As shown:

[0104] For the multi-scale CNN feature extraction module:

[0105] Three groups of 1D convolution kernels are used in parallel to extract features, such as Figure 4 As shown:

[0106] in:

[0107] It represents a one-dimensional convolution operation on the input x, with a convolution kernel length of k.

[0108] Represents the activation function. Here, the rectified linear unit (RectifiedLinearUnit) is used to introduce nonlinearity.

[0109] Wk: represents the convolution kernel (filter), whose length is k.

[0110] : Indicates the convolution operation.

[0111] x: represents the input data, usually a one-dimensional time series or text data.

[0112] bk: represents the bias term, corresponding to the convolution kernel Wk.

[0113] k∈{25,50,100}: indicates that the length of the convolution kernel can be 25, 50, or 100.

[0114] Note: The convolution kernel lengths correspond to the typical durations of the QRS complex (25 ms), ST segment (50 ms), and T wave (100 ms), respectively.

[0115] 25ms (12.5 sampling points): Captures the steep rising branch of the QRS wave (about 20-40ms).

[0116] 50ms (25 samples): matches the ST segment duration (usually 50-120ms).

[0117] 100ms (50 sampling points): covers the T wave width (100-200ms).

[0118]

[0119] To avoid the gradient vanishing of the deep network for small ST offsets (0.1mV), further residual connections are made:

[0120]

[0121] in:

[0122] Zout: represents the output tensor.

[0123] ReLU(·): Represents the Rectified Linear Unit (RectifiedLinearUnit) activation function, which is used to introduce nonlinearity.

[0124] Zconv: represents the output tensor after the convolution operation.

[0125] Zskip: represents the input tensor of the residual connection (skipconnection).

[0126] Zconv+Zskip: represents the element-wise addition of the convolution output and the residual input.

[0127] For the Transformer timing modeling module:

[0128] Linearly project the feature sequence (time step × channel) output by CNN into a 256-dimensional vector

[0129]

[0130] Introducing the self-attention mechanism of relative position encoding, such as Figure 5 As shown, the correlation between each time step is calculated to capture the dynamic evolution of the ST segment, and the correlation between each ECG band is automatically learned through the self-attention mechanism:

[0131] Multi-HeadAttention (MHA)

[0132]

[0133] in:

[0134] Q, K, V: represent query, key, and value matrices, respectively. They are all obtained by linear projection of the input matrix Z.

[0135] Softmax(·): The Softmax function is used to convert the attention scores into a probability distribution.

[0136] : Multiply the query matrix Q and the transpose of the key matrix K to get the attention score matrix.

[0137] : Scaling factor used to prevent the attention score from being too large and causing the Softmax gradient to disappear.

[0138] : Dimension of the key vector.

[0139] V: Value matrix, used to generate the final attention output.

[0140] Traditional sinusoidal position encoding may destroy the local periodicity of ECG. Relative Position Encoding, which is more adaptable to the heart rhythm, is used to preserve the waveform timing order:

[0141]

[0142] Added learnable lead embedding (LeadEmbedding), through cross-attention weighting, lead embedding

[0143] Interact with time features:

[0144]

[0145] in:

[0146] Zlead: represents the output after processing by the cross-attention mechanism.

[0147] CrossAttention(·): represents the cross-attention mechanism, which is usually used to handle the attention relationship between different modalities or different sequences.

[0148] L: represents an input matrix, usually a positional encoding or guidance matrix.

[0149] Z: represents another input matrix, usually a feature matrix.

[0150] For multi-task output header:

[0151] STEMI classification (Sigmoid output):

[0152] pSTEMI=σ(Wc·MeanPool(Z)+bc)

[0153] in:

[0154] pSTEMI: indicates the probability of STEMI (ST-segment elevation myocardial infarction).

[0155] σ(·): represents the activation function, usually the Sigmoid function, which is used to map the output to a probability range from 0 to 1.

[0156] Wc: represents the weight matrix, which is used to weight the input features.

[0157] MeanPool(Z): represents the average pooling operation on the matrix Z, which is usually used to reduce the dimension of the feature.

[0158] bc: represents the bias term, which is used to adjust the output baseline.

[0159] Based on clinical risk balance (cost of missed diagnosis > cost of misdiagnosis), the STEMI classification threshold was selected as 0.5, which can also be adjusted using the ROC curve.

[0160] ST segment deviation regression (lead by lead):

[0161]

[0162] ΔSTi: represents the ST segment change of the i-th lead.

[0163] Ws: represents the weight matrix, which is used to weight the input features.

[0164] : Represents the feature vector of the i-th lead, usually obtained from the cross-attention mechanism or other feature extraction methods.

[0165] bs: represents the bias term, which is used to adjust the output baseline.

[0166] i∈{1,…,12}: indicates that the formula applies to each of the 12 leads.

[0167] Outputs the ST deviation of 12 leads (unit: mm) to assist in determining whether the "adjacent lead elevation ≥ 1mm" standard is met.

[0168] The model must simultaneously meet the following requirements: ① ST elevation ≥ 1 mm (0.1 mV); and ② elevation in at least two adjacent leads. The regression task forces the model to learn accurate ST segment quantification.

[0169] In this embodiment, the sliding window result is processed as follows:

[0170] Input: 10-second ECG segments with a step size of 5 seconds (50% overlap)

[0171] Alarm condition: pSTEMI>0.5 in 3 consecutive windows and ΔST≥1mm in at least 2 adjacent leads

[0172]

[0173] Hierarchical alarm strategy:

[0174]

[0175] Based on the heat map generated by Grad-CAM, doctors can quickly verify whether the model's focus areas are reasonable through the heat map:

[0176]

[0177] Among them, ReLU: activation function, stands for Rectified Linear Unit, which is used to introduce nonlinearity.

[0178] ∑k: sum all items with index k.

[0179] αk: Weight coefficient, used to weight each item, which is the mean gradient of the kth channel.

[0180] : represents the partial derivative of pSTEMI with respect to the variable Zk.

[0181] Zk: Input variable or feature, which is the last layer feature of CNN.

[0182] The beneficial effects of the above technical solution are: fusing CNN and Transformer to realize multi-scale feature extraction and time series modeling of ECG signals, capturing the dynamic evolution of the ST segment through the self-attention mechanism of Transformer, improving diagnostic accuracy, combining with the attention mechanism to accurately locate the abnormal characteristics of STEMI and NSTEMI, providing an anatomical basis for revascularization, and through sliding window processing and alarm strategies, quickly responding to high-risk patients and optimizing emergency procedures.

[0183] The present invention provides an automatic identification method for STEMI and NSTEMI based on multimodal deep learning and data enhancement processing, including:

[0184] Obtain a set of historical lead ECG signals, and input each historical lead ECG signal into the data enhancement model in turn to obtain an enhanced variable value based on each valley point and peak point in each historical lead ECG signal;

[0185] Obtaining a first average value of the enhanced variable value based on the peak point and a second average value of the enhanced variable value based on the valley point of each historical lead ECG signal, and constructing a first average vector and a second average vector;

[0186] Performing binary regression analysis on the first average vector and the second average vector to generate regression variable values;

[0187] performing aggregation processing on the first average vector and the second average vector in sequence to obtain a first aggregate distribution of the first average vector and a second aggregate distribution of the second average vector;

[0188] Filtering aggregate reference variable values ​​based on the first aggregate distribution and the second aggregate distribution;

[0189] Perform a global search for signal segments outside the normal range on the denoised 12-lead ECG signal;

[0190] Calculate the abnormal coefficient of each signal segment to obtain the abnormal variance;

[0191] If the abnormal variance is less than the preset variance, it is determined that all signal segments are in a stable state, and signal amplitudes of all signal segments are adjusted based on the regression variable value;

[0192] Otherwise, all signal segments are determined to be in an unstable state, and the first segment of the normal range closest to each signal segment is extracted;

[0193] Randomly select a variable value from the regression variable value and the aggregate reference variable value to adjust the signal amplitude of the first segment;

[0194] An enhanced processed signal is obtained according to the amplitude adjustment result.

[0195] In this embodiment, the historical lead ECG signal set is 1,000 12-lead ECG data sets (such as .edf format) obtained from a hospital database, covering normal cases, myocardial infarction cases, etc. Valley / peak points: Taking the QRS complex as an example, the R peak is the "peak point" (the highest voltage point), and the S wave trough is the "valley point" (the lowest voltage point). The Pan-Tompkins algorithm is used to detect characteristic points such as the R peak and S valley of the ECG signal, and the timestamps and amplitude values ​​of the peak and valley points are output.

[0196] The enhanced variable value is an amplitude increment (such as +0.2mV) generated by adjusting the amplitude of the R peak or valley point.

[0197] In this embodiment, the enhanced variable values ​​of all peak points in each historical lead ECG signal are averaged to obtain a first average value, and the second average value is obtained in a similar principle to the first average value, which will not be described in detail here.

[0198] First average vector = {the first average value of each historical lead ECG signal};

[0199] The second average vector={the second average value of each historical lead ECG signal}.

[0200] In this embodiment, random forest regression is used to fit a target variable (such as the quantitative value of a clinical diagnosis label) using the first and second mean vectors as input features. After training is completed, a new mean vector is input and the model outputs the regression variable value.

[0201] In this embodiment, a Gaussian mixture model is used to fit the first average vector and the second average vector to obtain a distribution law describing the peak point characteristics and a distribution law describing the valley point characteristics.

[0202] In this embodiment, an algorithm identifies signal segments outside the normal range, such as ST-segment elevation exceeding 0.2 mV (characteristic of myocardial infarction) or T-wave amplitude below the normal range (e.g., <0.1 mV). A normal ECG template is preset (e.g., extracted from healthy individuals), and dynamic time warping (DTW) is used to calculate the difference between the input signal and the template, identifying intervals where the difference exceeds a threshold. For each time point, a normal range is set for the signal amplitude (e.g., the normal ST-segment amplitude is [-0.1, 0.1] mV), and continuous intervals outside the range are traversed to detect.

[0203] In this embodiment, for each sampling point in the abnormal interval, its deviation from the normal range is calculated (e.g., (value at the corresponding time point - normal maximum value) / normal maximum value) as the abnormal coefficient. The abnormal variance is obtained using the variance calculation formula and is used to measure the degree of dispersion.

[0204] In this embodiment, the preset variance is set based on clinical experience (eg, 0.1). If the abnormal variance is <0.1, the abnormality is considered to be a "stable deviation" (eg, chronic ST segment changes).

[0205] The signal amplitude adjustment is, for example, to linearly adjust the amplitude of the abnormal interval, such as: x_new=x_old+regression variable value×adjustment factor, and the adjustment factor is set to 0.1.

[0206] In this embodiment, a search is performed before and after the abnormal interval to find the nearest normal signal segment (such as a normal T wave interval) as an adjustment reference, and one is randomly selected from the regression variable value and the aggregate reference variable value (such as reg_value=0.3 or ref_value=0.25). The amplitude of the normal segment is adjusted, and a sliding window is used to traverse before and after the abnormal interval to find the first continuous segment that meets the normal range (length > preset value, such as 100ms).

[0207] Random adjustment uses random.choice([reg_value,ref_value]) to select variable values ​​and adjust the amplitude of the normal segment (such as ), simulated data augmentation.

[0208] The beneficial effects of the above technical solution are: through peak and valley point enhancement, regression analysis, and aggregate distribution, the enhanced data generated by data enhancement that fits the physiological characteristics of the ECG signal is more clinically valuable, and stable / unstable abnormalities are distinguished, and regression variables and adjacent normal segments are used for adjustment respectively. Combined with the statistical characteristics of historical data (average vector, aggregate distribution), the enhanced signal is made more consistent with the distribution law of the real ECG, thereby improving the model training effect.

[0209] The present invention provides an automatic identification method for STEMI and NSTEMI based on multimodal deep learning, which relies on the first aggregate distribution and the second aggregate distribution to screen the aggregate reference variable value, including:

[0210] Determine the average value and the number of aggregations under each first aggregation in the first aggregation distribution, and select the average value A1 of the first aggregation corresponding to the maximum number of aggregations;

[0211] Determine the average value and the number of aggregations under each second aggregation in the second aggregation distribution, and select the average value A2 of the second aggregation corresponding to the maximum number of aggregations;

[0212] The average value A1 and the average value A2 are averaged to obtain an aggregate reference variable value.

[0213] In this embodiment, for the first aggregation distribution, for example, aggregation 1: average value a1, including 500 samples, aggregation 2: average value a2, including 300 samples, aggregation 3: average value a3, including 200 samples, at this time, the maximum number of aggregations is aggregation 1, and the average value A1 is a1, and the method for obtaining the average value A2 is similar to the method for obtaining the average value A1, which will not be repeated here.

[0214] The beneficial effect of the above technical solution is: by screening the average value of the "maximum aggregation number", the most typical characteristics of the peak and valley points of the ECG signal are captured, providing a concise reference standard that fits the real data for subsequent signal adjustment.

[0215] The present invention provides an automatic identification method for STEMI and NSTEMI based on multimodal deep learning, constructing a total loss function, including:

[0216] Constructing the cross entropy loss function ;

[0217] Where N represents the number of samples, and the number of samples of ST segment and non-ST segment is equal; Indicates the sample label of the i-th non-ST segment; represents the sample label of the i-th ST segment; represents the predicted probability that the i-th non-ST segment belongs to acute non-ST-segment elevation myocardial infarction; represents the predicted probability that the i-th ST segment belongs to acute ST-segment elevation myocardial infarction; lg represents the sign of the logarithmic function; 、 They represent the number of historical missed diagnoses of non-ST segment and ST segment based on N, respectively; 、 represent the error coefficients respectively;

[0218] Constructing contrast-constrained loss function ;

[0219]

[0220] in, Indicates the minimum interval threshold; Represents the ST segment feature under the Zth feature dimension Non-ST segment features The cosine distance of Represents feature dimension; represents the set of contrast constraints based on D;

[0221] Constructing a task-balanced loss function ;

[0222] in, Indicates the i-th ST segment The predicted entropy of Indicates the i-th non-ST segment The predicted entropy of Based on The binary crossover function of Based on The binary crossover function of 、 They represent the balance compensation coefficient respectively;

[0223] Constructing a total loss function based on the cross entropy loss function, the contrast constraint loss function, and the task balance loss function;

[0224]

[0225] in, 、 Represent the adjustment coefficients respectively.

[0226] In this embodiment, the number of ST segment and non-ST segment samples in the formula is equal (N is evenly divided) to avoid class imbalance causing the model to be biased towards the majority class;

[0227] The number of historical missed diagnoses (nfs, ns) and error coefficients (α1, α2) are introduced to give higher penalty weights to categories with high risk of missed diagnosis (such as STEMI with serious consequences of missed diagnosis).

[0228] The task of distinguishing between ST segment and non-ST segment allows the model to simultaneously learn to identify two types of myocardial infarction, which meets the core diagnostic needs of the clinical "STEMI vs NSTEMI".

[0229] Conventional cross entropy cannot adapt to the cost differences and uneven sample distribution of clinical misdiagnosis / missed diagnosis. L1, through customized transformation, aligns the loss function with the high-risk scenarios of myocardial infarction diagnosis, which is the basis for the model's accurate classification.

[0230] The cosine distance (d(Zs,Zfs)) is used to measure the difference between ST segment and non-ST segment features, forcing the model to learn the essential distinction between the two types of features (such as ST segment elevation in STEMI vs ST segment depression in NSTEMI).

[0231] The introduction of a minimum interval threshold (m) and a logarithmic constraint (ln(...)) increases the penalty when the feature difference is less than m, ensuring that the model learns "sufficiently large feature discrimination." L2, through contrast constraints, forces the model to focus on key clinical features (ST segments), avoiding misdiagnosis due to insufficient feature learning, which is the key to improving model robustness.

[0232] Introducing prediction entropy (U(x)) to measure the uncertainty of the model in classifying samples (the larger the entropy, the more difficult the sample is to classify);

[0233] Dynamically adjusting the loss weights through 1+exp(-U(x)) increases the loss ratio for difficult-to-classify samples (such as early MI and atypical ECGs), forcing the model to learn these samples. Balancing compensation coefficients (σ1 and σ2) adjust the loss ratio for ST-segment and non-ST-segment tasks to account for the sample difficulty differences between these two tasks. MI data contains many difficult-to-classify samples (such as asymptomatic MI and atypical ECGs), and the learning difficulty of the two tasks (STEMI and NSTEMI) differs. L3, through dynamic weights and compensation coefficients, ensures that the model does not discard any sample type, which is the core of improving the diagnosis rate of complex cases.

[0234] By integrating L1 (classification accuracy), L2 (feature discrimination), and L3 (sample balance) using adjustment coefficients (γ1 and γ2), the model simultaneously optimizes classification accuracy, feature quality, and coverage of complex cases, meeting the high clinical requirements for MI diagnosis models. A single loss function cannot meet the complex requirements of MI diagnosis (which require accuracy, stability, and coverage of rare cases). Lz, through the synergy of multiple losses, systematically improves the model's clinical value, making it an essential design for industrial-grade diagnostic models.

[0235] In this example, if the risk of missed diagnosis of NSTEMI is low (clinical intervention effect is good), Take 1.0; if the consequences of missed diagnosis are serious (such as some NSTEMI progresses rapidly), Take 1.5~2.0;

[0236] The mortality rate of missed diagnosis of STEMI is high, and the penalty should be significantly increased. Usually it is 3.0~5.0.

[0237] The value of m is between 0.5 and 1.5.

[0238] The value of is between 0.5 and 0.8. The value of 2 is between 0.2 and 0.5;

[0239] If the ST segment samples are difficult to classify (e.g., the features of early myocardial infarction are fuzzy), σ1 is set to 0.7-0.8 to allow the model to focus more on such tasks.

[0240] If the non-ST segment samples are numerous but simple, σ2 is set to 0.2~0.3 to reduce its weight.

[0241] The value of is between 0.1 and 0.3. The value is between 0.2 and 0.5.

[0242] To prioritize reducing the missed diagnosis rate of STEMI: increase α2 (e.g., from 3.0 to 5.0) and simultaneously increase γ2 (L3 weight) to allow the model to focus more on difficult ST segment samples.

[0243] To improve feature interpretability (doctors need to understand the ST segment area that the model focuses on): increase γ1 (L2 weight) from 0.2 to 0.3 to strengthen the feature distinction between ST segment and non-ST segment.

[0244] The beneficial effect of the above technical solution is that conventional loss functions cannot adapt to clinical problems such as high risk, imbalance, and difficult samples, but the precise transformation formula applied to myocardial infarction diagnosis scenarios fills the gap between general methods and clinical needs.

[0245] The present invention provides an automatic identification method for STEMI and NSTEMI based on multimodal deep learning, which performs sliding window processing on the abnormal features of STEMI and NSTEMI, including:

[0246] The abnormal features of STEMI and NSTEMI are split into abnormal dimensions, and combined with the control relationship with the time point, the abnormalities in each dimension are marked at the corresponding time points on the input 12-lead ECG signal to obtain the abnormal distribution in each dimension;

[0247] Construct an anomaly set for each time point to determine the number of anomaly dimensions and the degree of anomaly, and match the first label obtained from the quantity-degree-label comparison table to the corresponding time point;

[0248] Starting from the initial time point of the input 12-lead ECG signal, the label sequence under each window is obtained according to the sliding window method;

[0249] Analyze the abnormal continuity of the label sequence under each window, and randomly screen N1 groups of three consecutive sequences for abnormal continuity. Combined with the overall characteristics of each group of three consecutive sequences and the correlation between the local characteristics and the overall characteristics of each sequence, analyze the continuous change direction of the abnormal continuity under each group. The continuous change direction includes: consistent direction change, enhanced direction change, attenuated direction change, and chaotic direction change.

[0250] Count the occurrence frequency of each change direction, and select the change direction with the maximum frequency as the final change direction;

[0251] The window adjustment coefficient is determined depending on all abnormal continuities under the final change direction, and the sliding window is resized in combination with a preset step size, and sliding window processing is performed.

[0252] In this example, STEMI (acute ST-segment elevation myocardial infarction) features abnormal ST segment morphology and amplitude: The ST segment exhibits an "arched" elevation, with the amplitude exceeding the clinical threshold (≥0.1 mV in limb leads, ≥0.2 mV in chest leads, and ≥0.15 mV in leads V2-V3 for women). ST segment elevation also occurs synchronously in multiple leads within the same infarct-related region (e.g., leads V1-V4 for anterior wall infarction, leads II, III, and aVF for inferior wall infarction), demonstrating a "clustered lead abnormality." Temporal variations include: Sudden ST segment elevation, typically rising from normal levels within a short period of time (minutes to tens of minutes), exhibiting a "steep rise" temporal trend. Sudden ST segment elevation persists (generally ≥30 minutes) and does not recover rapidly with body position or brief interventions (e.g., nitroglycerin), demonstrating a "plateau" temporal pattern. Associated waveform abnormalities: The corresponding leads may be accompanied by Q wave formation (indicating myocardial necrosis), peaked T waves (hyperacute phase) or subsequent T wave inversion (subacute phase), forming a "complex abnormal morphology" with ST segment elevation.

[0253] In this example, the abnormal characteristics of NSTEMI (non-ST-segment elevation myocardial infarction) include: abnormal ST segment or T wave morphology and amplitude: the ST segment is horizontally or downslopingly depressed, with an amplitude ≥0.05 mV (≥0.1 mV in some leads) and a duration ≥1 minute; the T wave is symmetrically inverted (e.g., coronary T waves), typically with a depth ≥0.1 mV. In some cases, mild ST segment depression and T wave inversion coexist. Temporal characteristics: Gradual: Abnormalities often develop slowly (over hours to days), with ST segment depression or T wave inversion exhibiting gradual deepening or fluctuating changes (e.g., recurrence of abnormalities after a brief recovery); Non-persistent: Abnormalities may temporarily subside with relief of ischemia, but overall they fluctuate repeatedly, without the sudden surges typical of STEMI. Lead distribution characteristics: Abnormal leads are relatively dispersed (e.g., limited to leads II, III, aVF or leads V4-V6), with less occurrence of "clustered abnormalities" in multiple leads, and more likely to be accompanied by "mirror image changes in corresponding leads" (e.g., mild elevation in the contralateral lead of the ST-segment depressed lead).

[0254] In this embodiment, abnormal dimension splitting: the high-dimensional abnormal feature vector extracted by the hybrid neural network is split into multiple independent feature dimensions (such as ST segment elevation amplitude, T wave inversion depth, etc.), time point comparison relationship: the mapping relationship between each abnormal dimension and the ECG signal time series is established (such as ST segment elevation at the 3rd second), abnormal distribution: the start and end time and intensity change of each dimension abnormality are marked on the time axis, for example, STEMI abnormal dimension: ST segment elevation amplitude: V1-V4 leads in Elevation of 0.2mV at t=5s, elevation of 0.3mV at t=10s, number of elevated leads: simultaneous elevation in leads II, III, and aVF (clustered abnormalities), Q wave formation: pathological Q wave appears in lead V2 at t=15s, for example, abnormal dimensions of NSTEMI: ST segment depression amplitude: 0.1mV depression in leads V4-V6 at t=20s, 0.15mV depression at t=30s, T wave inversion depth: 0.2mV inversion in leads II, III, and aVF at t=25s.

[0255] Abnormal set: The set of all abnormal dimensions at a certain time point (e.g., {t=10s:[ST segment elevation 0.3mV, number of leads=3]}).

[0256] The number of abnormal dimensions is the number of abnormal dimensions included in the abnormal set (for example, STEMI may have three dimensions such as ST segment elevation and Q wave formation at the same time). The abnormality degree is the quantitative intensity of the abnormality in each dimension (for example, ST segment elevation of 0.3mV is more severe than 0.1mV). The number-degree-label comparison table is a predefined mapping table that matches disease labels according to the number and degree of abnormal dimensions.

[0257] Example of a STEMI control chart

[0258]

[0259] Example of a NSTEMI control chart

[0260]

[0261] In this embodiment, fuzzy logic is used to process boundary values ​​(eg, taking a weighted average of adjacent labels when the ST segment is elevated by 0.18 mV).

[0262] In this embodiment, a sliding window is a fixed-length time window (e.g., 5 seconds) that slides across the ECG signal at a preset step size (e.g., 1 second). The label sequence is an ordered set of labels for all time points within each window (e.g., [low confidence STEMI, medium confidence STEMI]). For example, window parameters include: window length = 10 seconds, step size = 2 seconds. Label sequence generation: Window 1 (t = 0-10 seconds): [Normal, Low Confidence STEMI, Low Confidence STEMI]; Window 2 (t = 2-12 seconds): [Low Confidence STEMI, Low Confidence STEMI, Medium Confidence STEMI]. Specifically, the dynamic window length can be set to 10 seconds initially and subsequently adjusted based on the direction of abnormal changes. The overlapping step size is set to a step size < window length (e.g., step size = 5 seconds, window length = 10 seconds) to ensure temporal continuity.

[0263] In this embodiment, abnormal continuity is a pattern of continuous occurrence of abnormal labels in the label sequence (eg, three consecutive windows are all "high confidence STEMI").

[0264] The overall feature is the statistical quantity (such as mean, standard deviation) of the abnormality degree at all time points in the window.

[0265] The local feature is the degree of abnormality at a single time point within the window.

[0266] Continuously changing direction:

[0267] Consistent change in direction: No significant change in the degree of abnormality (e.g., all three windows are "medium confidence STEMI").

[0268] Directional enhancement changes: increasing severity of abnormality (e.g., low → moderate → high confidence STEMI).

[0269] Directional attenuation changes: the severity of abnormality decreases (e.g., high → medium → low confidence STEMI).

[0270] Chaotic changes in direction: no clear trend (e.g., high → low → intermediate confidence STEMI).

[0271] For example, in the STEMI example:

[0272] Window 1 (t = 0 ~ 10s): average ST segment elevation 0.2mV (medium confidence).

[0273] Window 2 (t = 2 ~ 12s): average ST segment elevation 0.3mV (high confidence).

[0274] Window 3 (t = 4 ~ 14s): average ST segment elevation 0.4mV (high confidence).

[0275] Change direction: Direction enhancement change (both local features and overall features increase).

[0276] NSTEMI Example:

[0277] Window 1 (t = 0 ~ 10s): average ST segment depression 0.08mV (low confidence).

[0278] Window 2 (t = 2 ~ 12s): mean ST segment depression 0.12mV (medium confidence).

[0279] Window 3 (t = 4-14s): mean ST segment depression 0.11 mV (medium confidence).

[0280] Change direction: The direction changes consistently (local features fluctuate, but overall features remain stable).

[0281] Specifically: Trend detection: use linear regression to fit the slope of the change in abnormality (slope > 0 indicates enhancement, ≈ 0 indicates consistency, and < 0 indicates attenuation); Correlation analysis: calculate the Pearson correlation coefficient between local features and overall features (r > 0.7 indicates strong correlation, supporting consistent / enhanced changes).

[0282] Frequency statistics are to count the number of times each change direction appears in all randomly screened N1 groups.

[0283] The final change direction is the change direction with the highest frequency (e.g., the enhanced change direction occurs 50 times, while other directions occur <30 times). The window adjustment coefficient is the coefficient that dynamically adjusts the window size based on the final change direction (e.g., when the enhanced change occurs, the coefficient = 0.5, and the window is reduced).

[0284] For example, in the STEMI scenario, three consecutive windows (N1 = 100) were randomly selected, and the directional enhancement change occurred 60 times. The final change direction was: directional enhancement change → window adjustment coefficient = 0.5 → window length was adjusted from 10s to 5s.

[0285] NSTEMI scenario: Consistent directional changes occurred 70 times. Final change direction: Consistent directional change → Window adjustment coefficient = 1.5 → Window length adjusted from 10s to 15s.

[0286] Specifically: Coefficient mapping table:

[0287]

[0288] The beneficial effects of the above technical solution are: through dimensional splitting and time labeling, abnormality positioning is refined from "overall signal" to "lead + time point", improving the diagnostic granularity, and the sliding window is combined with change direction analysis to allow the model to automatically adapt to the "development trend" of the abnormality (such as the gradual elevation of the ST segment in myocardial infarction), avoiding missed judgments in fixed windows, and label matching and direction analysis assist doctors in quickly identifying high-risk cases.

[0289] The present invention provides a method for automatically identifying STEMI and NSTEMI based on multimodal deep learning. After verifying whether the model's focus area is reasonable through the heat map, the method further includes:

[0290] After verifying that the model's focus area is unreasonable through the heat map, all offsets of the ST segment of the 12-lead ECG signal are obtained;

[0291] Performing a global analysis on all offsets using a global offset analysis model to obtain corresponding first and second variations;

[0292] Performing mixed analysis on all offsets using a mixed offset analysis model to obtain corresponding third and fourth variations;

[0293] Obtaining a feature loss function based on the first change amount, the second change amount, the third change amount, and the fourth change amount;

[0294] The optimized neural network is optimized again using the feature loss function.

[0295] In this embodiment, the global offset analysis model is a model built using LSTM+attention. The input is a sequence of ST segment offsets of a 12-lead ECG signal (e.g., [0.1mV, 0.2mV, -0.1mV, ...]) (the input dimension is an offset matrix of (time step, number of leads). The model learns the global pattern of the offsets (e.g., overall trend, coordinated changes in leads) and outputs two key changes (which can be mapped to scalars or vectors through a fully connected layer).

[0296] Offset: The offset value of the ST segment at each lead and each time point (e.g., the offset at the 100th sampling point in lead V1 is 0.3 mV).

[0297] First / second change: global feature changes in the model output, such as the "rate of change of the overall ST segment elevation trend" (first change) and the "amplitude of change in the offset difference between leads" (second change).

[0298] Hybrid offset analysis model: uses a CNN+Transformer hybrid architecture to simultaneously capture the local morphology of the offset (such as the ST segment curve of a single lead) and global correlation (such as the coordinated changes of multiple leads) (fusion of CNN (extracting local features) and Transformer (extracting global correlation). The input is also the offset matrix. CNN extracts local features at each time point (such as Conv1d processing the timing of each lead). Transformer learns global correlation and ultimately outputs two changes).

[0299] Third / fourth variation: changes in the output mixed features, such as "the degree of change in local ST segment morphological abnormality" (third variation) and "the change in the correlation strength of global-local features" (fourth variation).

[0300] In this embodiment, the loss function constraint model is designed by integrating four variables to focus on the reasonable area:

[0301] ,in, 、 、 、 They are the first change amount, the second change amount, the third change amount, and the fourth change amount respectively; 、 、 、 They are respectively the corresponding reasonable changes in clinical practice.

[0302] Use the feature loss function for backpropagation to adjust the model parameters so that the area the model focuses on is more in line with clinical practice (such as forcing the model to learn the reasonable change trend of the ST segment) to achieve further optimization.

[0303] The beneficial effects of the above technical solution are: after discovering that the model's focus area is unreasonable through the heat map, a finer-grained feature loss optimization is used to form a closed loop of "finding the problem → analyzing the problem → solving the problem"; the global and mixed offset analysis model converts clinical prior knowledge such as "ST segment change trend" and "lead synergy relationship" into an optimizable loss function, forcing the model to learn reasonable feature patterns, which can significantly improve diagnostic reliability.

[0304] The present invention provides an automatic identification system for STEMI and NSTEMI based on multimodal deep learning, such as Figure 2 Shown, including:

[0305] Constructing a building module for building a hybrid neural network architecture, wherein the hybrid neural network architecture consists of an input layer, a multi-scale CNN feature extraction module, a Transformer time series modeling module, and a multi-task output head;

[0306] a framework optimization module for optimizing the hybrid neural network framework based on a total loss function to obtain an optimized neural network, wherein ST segment samples of the total loss function correspond to STEMI labels and non-ST segment samples correspond to NSTEMI labels;

[0307] A feature acquisition module is used to input the 12-lead ECG signal into the input layer of the optimized neural network for preprocessing, and sequentially obtain abnormal features of STEMI and NSTEMI based on the multi-scale CNN feature extraction module, the Transformer time series modeling module, and the multi-task output head;

[0308] The alarm output module is used to perform sliding window processing on the abnormal features of STEMI and NSTEMI, and output alarms based on a hierarchical alarm strategy. At the same time, a heat map is generated based on Grad-CAM for doctors to verify whether the model's focus area is reasonable through the heat map.

[0309] The beneficial effects of the above technical solution are: fusing CNN and Transformer to realize multi-scale feature extraction and time series modeling of ECG signals, capturing the dynamic evolution of the ST segment through the self-attention mechanism of Transformer, improving diagnostic accuracy, combining with the attention mechanism to accurately locate the abnormal characteristics of STEMI and NSTEMI, providing an anatomical basis for revascularization, and through sliding window processing and alarm strategies, quickly responding to high-risk patients and optimizing emergency procedures.

[0310] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for automatic identification of STEMI and NSTEMI based on multimodal deep learning, characterized by: include: Step 1: Build a hybrid neural network architecture, where the hybrid neural network architecture consists of an input layer, a multi-scale CNN feature extraction module, a Transformer time series modeling module, and a multi-task output head; Step 2: Optimizing the hybrid neural network architecture based on the total loss function to obtain an optimized neural network, wherein the ST segment samples of the total loss function correspond to STEMI labels and the non-ST segment samples correspond to NSTEMI labels; Step 3: The 12-lead ECG signal is input into the input layer of the optimized neural network for preprocessing, and the abnormal features of STEMI and NSTEMI are obtained based on the multi-scale CNN feature extraction module, the Transformer time series modeling module, and the multi-task output head. Step 4: Perform sliding window processing on the abnormal features of STEMI and NSTEMI, and output alarms based on a hierarchical alarm strategy. At the same time, a heat map is generated based on Grad-CAM for doctors to verify whether the model's focus area is reasonable. Among them, the abnormal features of STEMI and NSTEMI are processed by sliding window, including: The abnormal features of STEMI and NSTEMI are split into abnormal dimensions, and combined with the control relationship with the time point, the abnormalities in each dimension are marked at the corresponding time points on the input 12-lead ECG signal to obtain the abnormal distribution in each dimension; Construct an anomaly set for each time point to determine the number of anomaly dimensions and the degree of anomaly, and match the first label obtained from the quantity-degree-label comparison table to the corresponding time point; Starting from the initial time point of the input 12-lead ECG signal, the label sequence under each window is obtained according to the sliding window method; Analyze the abnormal continuity of the label sequence under each window, and randomly screen N1 groups of three consecutive sequences for abnormal continuity. Combined with the overall characteristics of each group of three consecutive sequences and the correlation between the local characteristics and the overall characteristics of each sequence, analyze the continuous change direction of the abnormal continuity under each group. The continuous change direction includes: consistent direction change, enhanced direction change, attenuated direction change, and chaotic direction change. Count the occurrence frequency of each change direction, and select the change direction with the maximum frequency as the final change direction; The window adjustment coefficient is determined depending on all abnormal continuities under the final change direction, and the sliding window is resized in combination with a preset step size, and sliding window processing is performed.

2. The method for automatic identification of STEMI and NSTEMI based on multimodal deep learning according to claim 1, characterized in that: The preprocessing includes: lead-specific normalization processing, wavelet threshold-based denoising processing, and data enhancement processing.

3. The method for automatic identification of STEMI and NSTEMI based on multimodal deep learning according to claim 2, characterized in that: Data enhancement processing, including: Obtain a set of historical lead ECG signals, and input each historical lead ECG signal into the data enhancement model in turn to obtain an enhanced variable value based on each valley point and peak point in each historical lead ECG signal; Obtaining a first average value of the enhanced variable value based on the peak point and a second average value of the enhanced variable value based on the valley point of each historical lead ECG signal, and constructing a first average vector and a second average vector; Performing binary regression analysis on the first average vector and the second average vector to generate regression variable values; performing aggregation processing on the first average vector and the second average vector in sequence to obtain a first aggregate distribution of the first average vector and a second aggregate distribution of the second average vector; Filtering aggregate reference variable values ​​based on the first aggregate distribution and the second aggregate distribution; Perform a global search for signal segments outside the normal range on the denoised 12-lead ECG signal; Calculate the abnormal coefficient of each signal segment to obtain the abnormal variance; If the abnormal variance is less than the preset variance, it is determined that all signal segments are in a stable state, and signal amplitudes of all signal segments are adjusted based on the regression variable value; Otherwise, all signal segments are determined to be in an unstable state, and the first segment of the normal range closest to each signal segment is extracted; Randomly select a variable value from the regression variable value and the aggregate reference variable value to adjust the signal amplitude of the first segment; An enhanced processed signal is obtained according to the amplitude adjustment result.

4. The method for automatic identification of STEMI and NSTEMI based on multimodal deep learning according to claim 3, characterized in that: Filtering the aggregate reference variable value based on the first aggregate distribution and the second aggregate distribution includes: Determine the average value and the number of aggregations under each first aggregation in the first aggregation distribution, and select the average value A1 of the first aggregation corresponding to the maximum number of aggregations; Determine the average value and the number of aggregations under each second aggregation in the second aggregation distribution, and select the average value A2 of the second aggregation corresponding to the maximum number of aggregations; The average value A1 and the average value A2 are averaged to obtain an aggregate reference variable value.

5. The method for automatic identification of STEMI and NSTEMI based on multimodal deep learning according to claim 1, characterized in that: Construct the total loss function, including: Constructing the cross entropy loss function ; Where N represents the number of samples, and the number of samples of ST segment and non-ST segment is equal; Indicates the sample label of the i-th non-ST segment; represents the sample label of the i-th ST segment; represents the predicted probability that the i-th non-ST segment belongs to acute non-ST-segment elevation myocardial infarction; represents the predicted probability that the i-th ST segment belongs to acute ST-segment elevation myocardial infarction; lg represents the sign of the logarithmic function; 、 They represent the number of historical missed diagnoses of non-ST segment and ST segment based on N, respectively; 、 represent the error coefficients respectively; Constructing contrast-constrained loss function ; in, Indicates the minimum interval threshold; Represents the ST segment feature under the Zth feature dimension Non-ST segment features The cosine distance of Represents feature dimension; represents the set of contrast constraints based on D; Constructing task-balanced loss function ; in, Indicates the i-th ST segment The predicted entropy of Indicates the i-th non-ST segment The predicted entropy of Based on The binary crossover function of Based on The binary crossover function of 、 They represent the balance compensation coefficient respectively; Constructing a total loss function based on the cross entropy loss function, the contrast constraint loss function, and the task balance loss function; in, 、 Represent the adjustment coefficients respectively.

6. The method for automatic identification of STEMI and NSTEMI based on multimodal deep learning according to claim 1, characterized in that: After verifying whether the model's focus area is reasonable through heat maps, it also includes: After verifying that the model's focus area is unreasonable through the heat map, all offsets of the ST segment of the 12-lead ECG signal are obtained; Perform a global analysis on all offsets using a global offset analysis model to obtain the corresponding first and second variations; Performing mixed analysis on all offsets using a mixed offset analysis model to obtain corresponding third and fourth variations; Obtaining a feature loss function based on the first change amount, the second change amount, the third change amount, and the fourth change amount; The optimized neural network is optimized again using the feature loss function.

7. An automatic identification system for STEMI and NSTEMI based on multimodal deep learning, characterized by: include: Constructing a building module for building a hybrid neural network architecture, wherein the hybrid neural network architecture consists of an input layer, a multi-scale CNN feature extraction module, a Transformer time series modeling module, and a multi-task output head; a framework optimization module for optimizing the hybrid neural network framework based on a total loss function to obtain an optimized neural network, wherein ST segment samples of the total loss function correspond to STEMI labels and non-ST segment samples correspond to NSTEMI labels; A feature acquisition module is used to input the 12-lead ECG signal into the input layer of the optimized neural network for preprocessing, and sequentially obtain abnormal features of STEMI and NSTEMI based on the multi-scale CNN feature extraction module, the Transformer time series modeling module, and the multi-task output head; The alarm output module is used to perform sliding window processing on abnormal features of STEMI and NSTEMI, and output alarms based on a hierarchical alarm strategy. At the same time, a heat map is generated based on Grad-CAM for doctors to verify whether the model's focus area is reasonable through the heat map; Among them, the abnormal features of STEMI and NSTEMI are processed by sliding window, including: The abnormal features of STEMI and NSTEMI are split into abnormal dimensions, and combined with the control relationship with the time point, the abnormalities in each dimension are marked at the corresponding time points on the input 12-lead ECG signal to obtain the abnormal distribution in each dimension; Construct an anomaly set for each time point to determine the number of anomaly dimensions and the degree of anomaly, and match the first label obtained from the quantity-degree-label comparison table to the corresponding time point; Starting from the initial time point of the input 12-lead ECG signal, the label sequence under each window is obtained according to the sliding window method; Analyze the abnormal continuity of the label sequence under each window, and randomly screen N1 groups of three consecutive sequences for abnormal continuity. Combined with the overall characteristics of each group of three consecutive sequences and the correlation between the local characteristics and the overall characteristics of each sequence, analyze the continuous change direction of the abnormal continuity under each group. The continuous change direction includes: consistent direction change, enhanced direction change, attenuated direction change, and chaotic direction change. Count the occurrence frequency of each change direction, and select the change direction with the maximum frequency as the final change direction; The window adjustment coefficient is determined depending on all abnormal continuities under the final change direction, and the sliding window is resized in combination with a preset step size, and sliding window processing is performed.

Images