Myocardial infarction positioning method based on cross-modal knowledge fusion

Abstract

The invention provides a myocardial infarction positioning method based on cross-modal knowledge fusion. The method comprises the following steps: acquiring a target 1D-VCG signal and constructing a 2D-VCG image data set; extracting spatial features, time sequence features and pseudo tags from the 2D-VCG signals and the 1D-VCG signals through a teacher model and a student model; aligning and transforming the spatial features and the time sequence features through a cross-modal generator; and cross-modal knowledge fusion and MI positioning are realized through a cross-modal generator. According to the method, through knowledge enhancement and student enhancement strategies, the accuracy of myocardial infarction positioning is improved, and the overall framework of cross-modal learning is optimized through the adaptive weight method; the method is realized in a knowledge distillation form, spatial features of 2D-VCG are successfully fused into 1D-VCG time sequence analysis, and the positioning accuracy of myocardial infarction is remarkably improved by combining cooperative work of a teacher model, a student model and a cross-modal generator.

Description

Technical Field

[0001] The present invention relates to the technical field of myocardial infarction positioning, and in particular to a myocardial infarction positioning method based on cross-modal knowledge fusion. Background Art

[0002] Myocardial infarction (MI) is a serious heart disease that can cause rapid and irreversible damage to the myocardium. Timely diagnosis and treatment are crucial to prevent the disease from worsening.

[0003] As non-invasive detection methods, 12-lead electrocardiogram (ECG) and 3-lead vectorcardiogram (VCG) are often used to locate MI in clinical practice. Given the high readability and wide application of 12-lead ECG, some studies have analyzed 12-lead ECG based on artificial intelligence methods to locate MI. Although the use of ECG has achieved positive results and confirmed the possibility of computer intelligent assisted MI positioning, its advantages in the positioning of certain types of MI still need to be improved. For example, in the diagnosis of acute myocardial infarction in the posterior wall, the use of VCG can show better diagnostic performance. This is partly because the ECG is a secondary projection of VCG in the frontal plane and the horizontal plane, which cannot fully capture the spatial information of cardiac electrical activity.

[0004] In this regard, artificial intelligence algorithm analysis of the electrocardiogram obtained by monitoring cardiac activity with the Frank orthogonal lead system has become a major research trend in the development of computer intelligent assisted positioning MI systems in recent years. Given that the electrocardiogram is a three-dimensional representation of electrical activity composed of signals from three orthogonal leads, it depicts the direction and intensity of the heart's electrical activity at each time point. It is a vector representation of cardiac activity in space. In the field of artificial intelligence-based cardiac electrical activity analysis, the models developed are usually for one-dimensional time series or two-dimensional images. In this regard, existing research on VCG usually establishes a one-dimensional machine learning model to obtain time series features or spectral features of three orthogonal leads, or establishes a two-dimensional deep learning architecture to obtain spatial features of the projection of VCG on the three orthogonal planes of the frontal plane, horizontal plane and sagittal plane, so as to locate myocardial infarction.

[0005] Although the above research has made remarkable progress in recent years, the paradigm of learning myocardial infarction localization representation from a single modality has gradually encountered bottlenecks. Among them, extracting nonlinear features from one-dimensional vectorcardiograms can be complex and cumbersome, and fails to optimally utilize the spatial and inter-planar correlations of the vectorcardiogram. Secondly, for 2D plane projections, as the duration of the vectorcardiogram signal increases, its internal spatial structure becomes more complex, and the 2D projection may miss the subtle details inherent in the 3D vectorcardiogram, which poses a major challenge to myocardial infarction localization that prioritizes vectorcardiogram segments rather than obvious vectorcardiogram beats and the lightweight deployment of models at the edge. Summary of the invention

[0006] In view of the deficiencies of the prior art, the present invention provides a myocardial infarction localization method based on cross-modal knowledge fusion, which makes full use of the richer spatial features contained in the projection images of VCG on three orthogonal planes.

[0007] The technical solution of the present invention is: a method for locating myocardial infarction based on cross-modal knowledge fusion, comprising the following steps:

[0008] S1), collecting target 1D-VCG signals and constructing a 2D-VCG image dataset, and then dividing the 2D-VCG image dataset into a training set and a test set;

[0009] S2), constructing a teacher model and training it, and extracting spatial features and pseudo-label Logits of the 2D-VCG image through the teacher model;

[0010] S3), constructing a student model to train it, and obtaining timing features and pseudo-label Logits from the 1D-VCG signal through the student model;

[0011] S4), construct a cross-modal generator, align and transform the spatial features extracted by the teacher model and the temporal features extracted by the student model through the cross-modal generator, and realize cross-modal knowledge fusion and MI positioning through the cross-modal generator.

[0012] Preferably, in step S2), the teacher model is iteratively optimized based on a graph convolutional neural network (MVCNN), and the teacher model obtains features of each view by using multiple parallel ResNet18 architectures; and the core spatial features of each view are intercepted and weightedly aggregated through a multi-view collaborative attention fusion module.

[0013] Preferably, in step S2), the multi-view collaborative attention fusion module includes a two-stage attention mechanism, and the semantic information of feature maps from different perspectives is enhanced through the two-stage attention mechanism.

[0014] Preferably, in step S2), relevant features of different perspectives are aggregated through a stage of attention mechanism, specifically:

[0015] S211), the feature tensors (f i 1 , f i 2 , f i 3 ) compresses the channel features through the convolution layer and combines them into a tensor F i Ang; The calculation expression is:

[0016] f i Ang =ReLU(Conv(Cat(f i 1 ,f i 2 ,f i 3 )))

[0017] In the formula, (f i 1 ,f i 2 ,f i 3 ) are tensors of three different perspectives, and Cat represents the tensor views of three different perspectives (f i 1 ,f i 2 ,f i 3 ) are connected in series along the channel; Conv represents the convolution operation; ReLU is the activation function; the subscript i represents the index of the sample currently being processed, which is used to distinguish different samples;

[0018] S212), calculate the attention factor through the sigmoid function; that is:

[0019] u A =σ(f i Ang ×1U A1 ×2U A2 ×3U A3 )

[0020] u B =σ(f i Ang ×1U B1 ×2U B2 ×3U B3 )

[0021] u C =σ(f i Ang ×1U C1 ×2U C2 ×3U C3 )

[0022] In the formula, u A ,u B ,u C is the attention factor; represents the feature weights at different perspectives, which is calculated using the sigmoid function to adjust the importance of each feature at different perspectives; σ represents the sigmoid function; {UAi ,U Bi ,U Ci} represents the parameter matrix of the attention factor calculation module, which represents the weight allocation of different feature channels. Through these matrices, the model can capture the importance of feature tensors on different channels; the parameter matrix is ​​used for three different perspective feature channels; × represents modular multiplication; A, B, and C represent the labels of the three different perspective feature channels. A, B, and C are used to distinguish features from different perspectives, so that the attention factor can be independently weighted and fused for each perspective;

[0023] S213), by multiplying the attention factor by the corresponding feature tensor element by element, the enhanced feature tensor F is obtained i Ang ;Right now:

[0024] F i Ang =f i 1 ⊙u A +f i 2 ⊙u B +f i 3 ⊙u c}

[0025] In the formula, F i Ang Represents the feature tensor of the enhanced first-stage attention; ⊙ represents element-by-element multiplication;

[0026] Then two parallel internal attention feature fusion modules are used to enhance the semantic information of the feature map from different perspectives.

[0027] Preferably, in step S2), the attention mechanism of the second stage further interacts the feature tensors from the first stage, and multiple multi-view feature tensors complement each other to form a final output.

[0028] Preferably, in step S2), the second-stage attention mechanism processes the feature tensor of the first stage to obtain the final output, which specifically includes the following steps:

[0029] S221), through the maximum pooling and average pooling operations, obtain F i A1 and F i A2 ,Right now:

[0030] F i A1 =Conv(ReLU(Conv(AvgPool)(F iAng ))))

[0031] F i A2 =Conv(ReLU(Conv(MaxPool)(F i Ang ))))

[0032] In the formula, F i Ang Represents the feature tensor of the first stage of the enhanced attention mechanism, MaxPool and AvgPool represent the maximum pooling and average pooling operations respectively; Conv represents the convolution operation; ReLU is the activation function; F i A1 and F i A2 They represent the feature tensor after the maximum pooling MaxPool and average pooling AvgPool operations, respectively, to further extract the salient information of the features; MaxPool retains the main information of the feature map by taking the maximum value in the local window to highlight the salient features; AvgPool smoothes the feature map by calculating the average value of the local window to reduce the detail noise of the features;

[0033] S222), combined with the self-attention mechanism, and the final feature tensor F is generated through convolution operation i ′Ang ,Right now:

[0034] F i Ang =F i Ang ⊙σ(F i A1 +F i A2 )

[0035] F i ' A1 =avg(F i Ang )

[0036] F i ' A2 =max(F i Ang )

[0037] F i′Ang =F i Ang ⊙σ(Conv(Cat(F′ i A1 ,F′ i A2 )))

[0038] In the formula, max represents the average maximum value of elements along the channel dimension, Cat represents the concatenation of two feature tensors along the channel dimension to integrate the feature information of different pooling operations; avg represents the average value of elements along the channel dimension; F i ' A1 , F′ i A2 Respectively represent the feature tensor F i Ang The result after average pooling and maximum pooling operations; where F i A1 =avg(F i Ang ), which represents the feature tensor F i Ang The result of average pooling is used to smooth features; F i A2 =avg(F i Ang ) represents the feature tensor F i Ang The result of maximum pooling is used to extract significant features; ⊙ represents element-by-element multiplication; σ represents the sigmoid function; Conv represents the convolution operation.

[0039] Preferably, in step S3), the student model adopts the ResNet18 architecture as a feature extractor and myocardial infarction localization classifier to extract the time-frequency features of the 1D-VCG timing signal, and obtains the timing features and the output pseudo-labels through the maximum pooling layer and the fully connected layer of the student model, respectively, for use in the subsequent knowledge enhancement optimization and student enhancement optimization in the cross-modal knowledge distillation.

[0040] Preferably, in step S3), the student model can effectively extract and learn the temporal features related to myocardial infarction by combining the cross-modal distillation technology, and uses a deep convolutional network structure to capture the subtle timing changes of 1D-VCG data.

[0041] Preferably, in step S4), the cross-modal generator adopts a linear transformation strategy to ensure that features from different modalities are accurately mapped and aligned in the Euclidean space.

[0042] Preferably, in step S4), the cross-modal generator transforms the spatial feature f of the 2D-VCG through a parameterized transformation matrix W. 2D Converted to the timing feature f of 1D-VCG 1D Alignment form:

[0043] f align =W·f 2D +b

[0044] Where W is the conversion matrix; b is the bias term; f align is the transformed feature.

[0045] Preferably, in step S4), the cross-modal generator uses a variational autoencoder VAE to further optimize the features after feature alignment. The variational autoencoder VAE reconstructs a new feature representation consistent with the temporal feature space while ensuring the information richness of the features and adaptability to unseen data, and optimizes the following objective function by minimizing the reconstruction error and KL divergence:

[0046]

[0047] in, represents the knowledge enhancement loss of the variational autoencoder (VAE), which is used to ensure that the feature f align In the alignment process, necessary information is retained and unseen data is adapted. Encoder and Decoder represent variational autoencoder VAE and decoder respectively. The encoder converts the aligned features f align Mapped to the latent space, the decoder maps it back to the original feature space to achieve feature reconstruction; KL stands for Kullback-Leibler divergence, which is used to measure the difference between the learned distribution and the standard normal distribution. This item encourages the distribution of the latent space to be close to the normal distribution. Thus regularization is achieved; β is a hyperparameter that adjusts the importance of KL divergence, controlling the balance between reconstruction error and distribution matching. A larger β value will pay more attention to distribution matching, while a smaller value will pay more attention to reconstruction accuracy; f 1D Represents the timing characteristics of 1D-VCG, which is used to calculate the reconstruction error; Represents the standard normal distribution as a target for the latent space distribution, helping to optimize the latent representation of the VAE.

[0048] Preferably, in step S5), the cross-modal knowledge fusion includes knowledge enhancement optimization and student enhancement optimization.

[0049] Preferably, in step S5), the knowledge enhancement optimization refers to converting the spatial features of the cross-modal image obtained in the 2D-VCG into the Euclidean space, and independently transforming the features obtained by the teacher model and the student model into spatiotemporal features through a point generator, and performing loss supervision through a variational autoencoder to obtain a knowledge enhancement loss

[0050] Preferably, in step S5), the student enhancement optimization comprises the following steps:

[0051] S51), input 1D-VCG into the encoder of the student model to generate a temporal feature representation fts , and further optimized through a feature mapping layer U:

[0052] f optimized =Uf ts

[0053] In the formula, f optimized is the optimized feature of the feature mapping layer; U is the feature mapping layer; f ts Generate temporal feature representation;

[0054] S52), the optimized feature f optimized Used to train the classifier and generate category prediction P pred ; and predict P by category pred Calculating cross entropy loss

[0055]

[0056] In the formula, CrossEntropy represents the cross entropy loss function, which is used to calculate the loss between the predicted result and the true label in the classification task, and y true Represents the true label, which is used to compare with the model prediction results; P pred Represents the category prediction result of the model, which is used to calculate the loss with the true label;

[0057] S53), use distillation loss To optimize:

[0058]

[0059] Where P teacher is the output of the teacher model; λ is a trade-off factor used to adjust the weight between the cross entropy loss and the MSE loss; MSE stands for Mean Squared Error Loss, which is used to calculate the difference between the student model prediction result and the teacher model output to help the student model better imitate the teacher model;

[0060] S54), the final loss of the cross-modal generator is:

[0061]

[0062] In the formula, α, β, and γ represent the weight coefficients of different loss terms, which are used to adjust the influence of each loss term in the final loss function; α corresponds to the knowledge enhancement loss The weight of β corresponds to the cross entropy loss. The weight of γ corresponds to the distillation loss The weight of Enhance loss for knowledge; is the cross entropy loss; is the distillation loss.

[0063] The beneficial effects of the present invention are:

[0064] 1. The present invention enhances the orthogonal lead signal analysis of VCG by utilizing the visual image of 2D-VCG, so as to effectively obtain auxiliary knowledge from the view image;

[0065] 2. This paper develops a teacher-student framework and formulates cross-modal learning as a knowledge distillation problem. The new classifier enhancement criterion eliminates the distribution differences between different modalities and effectively avoids potential negative transfer.

[0066] 3. The present invention not only improves the accuracy of myocardial infarction localization through knowledge enhancement and student enhancement strategies, but also optimizes the overall framework of cross-modal learning through the adaptive weight method;

[0067] 4. The present invention is implemented in the form of knowledge distillation, successfully integrating the spatial features of 2D-VCG into the 1D-VCG temporal analysis, combining the collaborative work of the teacher model, the student model and the cross-modal generator, and significantly improving the localization accuracy of myocardial infarction;

[0068] 5. The present invention adopts a multi-view convolutional neural network as a teacher model, and weights and merges image features from different perspectives through a collaborative attention fusion module. This strategy enhances the model's ability to capture the spatial heterogeneity of cardiac activity and improves the semantic richness and recognition accuracy of features;

[0069] 6. The cross-modal generator of the present invention achieves precise alignment of data of different modalities through a concise and effective linear or convolutional layer configuration combined with variational autoencoder supervision; it not only optimizes the consistency at the feature level, but also avoids potential negative migration due to modality differences, ensuring the efficiency and accuracy of data integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the framework of the method of the present invention;

[0071] Figure 2 It is a framework diagram of the teacher model of the present invention;

[0072] Figure 3 It is a framework diagram of the student model of the present invention;

[0073] Figure 4 This is a framework diagram of the cross-modal generator of the present invention. DETAILED DESCRIPTION

[0074] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0075] like Figure 1 As shown, this embodiment provides a method for locating myocardial infarction based on cross-modal knowledge fusion, comprising the following steps:

[0076] S1), constructing a 2D-VCG image dataset, and dividing the 2D-VCG image dataset into a training set and a test set;

[0077] This embodiment uses a vectorcardiograph or a high-density body surface potential acquisition device to acquire target 1D-VCG signals, and acquires 2D-VCG images through downsampling and filtering, as well as projection mapping.

[0078] In this embodiment, the vectorcardiogram data set of myocardial infarction patients and normal persons provided by Physikalisch-Technische Bundesanstalt (PTB) is used; the original sampling frequency of the data is 1000 Hz.

[0079] The downsampling is to downsample the sampling frequency to 512 Hz to reduce the computational complexity; and the filtering adopts a second-order Butterworth high-pass filter with a cutoff frequency of 0.5 Hz to eliminate baseline drift;

[0080] Timing signal acquisition: The Pan-Tompkins algorithm was used to detect the R wave, and a non-overlapping 1D-VCG timing signal of 0.75 seconds was obtained by intercepting 0.25 seconds to the left and 0.75 seconds to the right with the R peak as the center;

[0081] Image signal acquisition: 2D-VCG images were acquired by projecting the vectorcardiogram on three orthogonal planes: frontal, horizontal, and sagittal planes. Each image was non-overlapping and consistent with the timing signal, consisting of a single projection image consisting of 0.75 seconds.

[0082] S2), constructing a teacher model and training it, and extracting spatial features of the 2D-VCG image through the teacher model;

[0083] In this embodiment, Figure 2 As shown, the teacher model adopts the graph convolutional neural network MVCNN, and the teacher model obtains the features of each view by multiple parallel ResNet18 architectures; and the teacher model intercepts and aggregates the core spatial features of each view through a multi-view collaborative attention fusion module.

[0084] The multi-view collaborative attention fusion module includes a two-stage attention mechanism, which enhances the semantic information of feature maps from different perspectives for spatial feature and pseudo-label acquisition.

[0085] Among them, the relevant features of different perspectives are aggregated through the attention mechanism of the first stage, specifically:

[0086] S211), the feature tensors (f i 1 ,fi i 2 ,fi i 3 ) compresses the channel features through the convolution layer and combines them into a tensor F i Ang ; The calculation expression is:

[0087] f i Ang =ReLU(Conv(Cat(f i 1 ,f i 2 ,f i 3 )))

[0088] In the formula, (f i 1 ,f i 2 ,f i 3 ) are the feature tensors of three different perspectives, and Cat represents the feature tensors of three different perspectives (f i 1 ,f i 2 ,f i 3 ) are connected in series along the channel; Conv represents the convolution operation; ReLU is the activation function; the subscript i represents the index of the sample currently being processed, which is used to distinguish different samples;

[0089] S212), calculate the attention factor through the sigmoid function; that is:

[0090] u A =σ(f i Ang ×1U A1 ×2U A2 ×3U A3 )

[0091] u B =σ(f i Ang ×1U B1 ×2U B2 ×3U B3 )

[0092] u C =σ(fi Ang ×1U C1 ×2U C2 ×3U C3 )

[0093] In the formula, u A ,u B ,u C is the attention factor, which indicates the feature weight under different perspectives. It is calculated by using the sigmoid function, which is used to adjust the importance of each feature under different perspectives. σ indicates the sigmoid function; {U Ai ,U Bi ,U Ci} is the parameter matrix of the attention factor calculation module, which represents the weight allocation of different feature channels. Through these matrices, the model can capture the importance of feature tensors on different channels; the parameter matrix is ​​used for three different perspective feature channels; × represents modular multiplication; A, B, and C represent the labels of the three different perspective feature channels, respectively. A, B, and C are used to distinguish features from different perspectives, so that the attention factor can be independently weighted and fused for each perspective;

[0094] S213), by multiplying the attention factor by the corresponding feature tensor element by element, the enhanced feature tensor F is obtained i Ang ;Right now:

[0095] F i Ang =f i 1 ⊙u A +f i 2 ⊙u B +f i 3 ⊙u c}

[0096] In the formula, F i Ang Represents the feature tensor of the first stage of the enhanced attention mechanism; ⊙ represents element-by-element multiplication;

[0097] Then two parallel internal attention feature fusion modules are used to enhance the semantic information of the feature map from different perspectives.

[0098] In this embodiment, the attention mechanism of the second stage further interacts with the feature tensor from the attention mechanism of the first stage, and multiple multi-view feature tensors complement each other to form the final output, which specifically includes the following steps:

[0099] S221), through the maximum pooling and average pooling operations, obtain Fi A1 and F i A2 ,Right now:

[0100] F i A1 =Conv(ReLU(Conv(AvgPool)(F i Ang ))))

[0101] F i A2 =Conv(ReLU(Conv(MaxPool)(F i Ang ))))

[0102] In the formula, F i Ang Represents the feature tensor of the first stage of the enhanced attention mechanism, MaxPool and AvgPool represent the maximum pooling and average pooling operations respectively; Conv represents the convolution operation; ReLU is the activation function; F i A1 and F i A2 They represent the feature tensor after the maximum pooling MaxPool and average pooling AvgPool operations, respectively, to further extract the salient information of the features; MaxPool retains the main information of the feature map by taking the maximum value in the local window to highlight the salient features; AvgPool smoothes the feature map by calculating the average value of the local window to reduce the detail noise of the features;

[0103] S222), combined with the self-attention mechanism, and the final feature tensor F is generated through convolution operation i ′Ang ,Right now:

[0104] F i Ang =F i Ang ⊙σ(F i A1 +F i A2 )

[0105] F′ i A1 =avg(F i Ang )

[0106] F′ i A2 =max(F i Ang )

[0107] F′ i Ang =F i Ang ⊙σ(Conv(Cat(F′ i A1 ,F′ i A2 )))

[0108] In the formula, max represents the average maximum value of elements along the channel dimension, Cat represents the concatenation of two feature tensors along the channel dimension to integrate the feature information of different pooling operations; avg represents the average value of elements along the channel dimension; F′ i A1 , F′ i A2 Respectively represent the feature tensor F i Ang The result after average pooling and maximum pooling operations; where F i A1 =avg(F i Ang ), which represents the feature tensor F i Ang The result of average pooling is used to smooth features; F i A2 =avg(F i Ang ) represents the feature tensor F i Ang The result of the maximum pooling is used to extract significant features; ⊙ represents element-by-element multiplication, combining the feature tensor with the attention factor to adjust the feature weight; σ represents the sigmoid function, which is used to calculate the attention factor and help adjust the importance of different features; Conv represents the convolution operation, which is used to further extract and enhance feature information;

[0109] In addition, the dimension of the feature tensor of the teacher model in this embodiment is: 3 (number of channels) × 256 (height) × 256 (width); and the number of filters in the convolution layer is 32, and the convolution kernel size is set to 3×3 to ensure the extraction of rich feature information. The activation function uses ReLU, and the pooling size of the maximum pooling and average pooling is 2×2 to effectively reduce the size of the feature map. The initial learning rate is set to 0.001, the number of training rounds is 50, and the loss function combines the cross entropy loss (L CE ) and contrast loss (L CL ) to optimize model performance.

[0110] S3), constructing a student model to train it, and obtaining timing features from the 1D-VCG signal through the student model;

[0111] In this embodiment, Figure 3 As shown, the student model uses the ResNet18 architecture as a feature extractor and myocardial infarction localization classifier to extract the time-frequency features of the 1D-VCG timing signal, and obtains the timing features and the output pseudo-labels through the maximum pooling layer and the fully connected layer of the student model, respectively, for subsequent knowledge enhancement and student enhancement in cross-modal knowledge distillation.

[0112] The student model can effectively extract and learn the temporal features related to myocardial infarction by combining the cross-modal distillation technology. The student model adopts a deep convolutional network structure to capture the subtle temporal changes of 1D-VCG data.

[0113] The student model includes multiple convolutional layers and fully connected layers. The convolutional layer is configured with 16 filters of size 1×5 and stride 1. There is no padding to ensure accurate processing of the time dimension. The activation function uses ReLU to provide nonlinear processing capabilities. Batch normalization and maximum pooling are applied after each convolution, and the pooling size is 2×1 to enhance the generalization ability of the model.

[0114] And the student model uses a loss function optimized specifically for time series data, combined with the MSE and the soft label output of the teacher model to form a distillation loss

[0115]

[0116] Among them, α and λ are hyperparameters, and their optimal values ​​are determined by Bayesian optimization adaptive tuning; CrossEntropy represents the cross entropy loss function, which is used to calculate the loss between the predicted results and the true labels in the classification task, and y true Represents the true label, which is used to compare with the model prediction results; P pred Represents the category prediction result of the model, which is used to calculate the loss with the true label;

[0117] During the training process, the Adam optimizer was used and the initial learning rate was set to 0.0001 to ensure a stable learning progress.

[0118] S4) constructing a cross-modal generator, and aligning and transforming the spatial features extracted by the teacher model and the temporal features extracted by the student model through the cross-modal generator.

[0119] In this embodiment, Figure 4 As shown, the cross-modal generator transforms the spatial features f of 2D-VCG through a parameterized linear layer W 2D Converted to the timing feature f of 1D-VCG 1D Alignment form:

[0120] falign =W·f 2D +b

[0121] Where W is the conversion matrix; b is the bias term; f align is the transformed feature.

[0122] The cross-modal generator uses a variational autoencoder (VAE) to further optimize the features after feature alignment. The variational autoencoder (VAE) reconstructs a new feature representation consistent with the temporal feature space while ensuring the information richness of the features and adaptability to unseen data, and optimizes the knowledge enhancement loss by minimizing the reconstruction error and KL divergence.

[0123]

[0124] Where Encoder and Decoder represent the variational autoencoder VAE and decoder respectively, KL represents the Kullback-Leibler divergence, which is used to measure the difference between the learned distribution and the standard normal distribution, and β is a hyperparameter for adjusting the importance of KL divergence; represents the knowledge enhancement loss of the variational autoencoder (VAE), which is used to ensure that the feature f align In the alignment process, necessary information is retained and unseen data is adapted. Encoder and Decoder represent variational autoencoder VAE and decoder respectively. The encoder converts the aligned features f align Mapped to the latent space, the decoder maps it back to the original feature space to achieve feature reconstruction; KL stands for Kullback-Leibler divergence, which is used to measure the difference between the learned distribution and the standard normal distribution. This item encourages the distribution of the latent space to be close to the normal distribution. Thus regularization is achieved; β is a hyperparameter that adjusts the importance of KL divergence, controlling the balance between reconstruction error and distribution matching. A larger β value will pay more attention to distribution matching, while a smaller value will pay more attention to reconstruction accuracy; f 1D Represents the timing characteristics of 1D-VCG, which is used to calculate the reconstruction error; Represents the standard normal distribution as a target for the latent space distribution, helping to optimize the latent representation of the VAE;

[0125] S5), cross-modal knowledge fusion and MI positioning;

[0126] The spatial features extracted from 2D-VCG by the teacher model and the temporal features extracted from 1D-VCG by the student model are fused through a cross-modal generator to achieve high-precision localization of myocardial infarction.

[0127] The cross-modal knowledge fusion includes knowledge enhancement optimization and student enhancement optimization. The knowledge enhancement optimization refers to transforming the spatial features of the cross-modal image obtained in the 2D-VCG into the Euclidean space, and independently transforming the features obtained by the teacher model and the student model into spatiotemporal features through the point generator, and performing loss supervision through the variational autoencoder to obtain the knowledge enhancement loss Specifically:

[0128] The 2D-VCG image features extracted by the teacher model are first adjusted by a transformation matrix W to adapt to the 1D-VCG temporal feature space of the student model:

[0129] f align =W·f 2D +b

[0130] Where W is the transformation matrix and b is the bias term. These parameters are learned through training to optimize the spatial alignment of features.

[0131] The transformed feature f align Further optimization is performed through a specially designed variational autoencoder VAE; the loss function of VAE includes reconstruction loss and KL divergence, and the formula is:

[0132]

[0133] The student enhancement optimization comprises the following steps:

[0134] S51), input 1D-VCG into the encoder of the student model to generate a temporal feature representation f ts , and further optimized through a feature mapping layer U:

[0135] f optimized =Uf ts

[0136] In the formula, f optimized is the optimized feature of the feature mapping layer; U is the feature mapping layer; f ts Generate temporal feature representation;

[0137] S52), the optimized feature f optimized Used to train the classifier and generate category prediction P pred ; and predict P by category pred Calculating cross entropy loss

[0138]

[0139] In the formula, CrossEntropy represents the cross entropy loss function, which is used to calculate the loss between the predicted result and the true label in the classification task, and ytrue Represents the true label, which is used to compare with the model prediction results; P pred Represents the category prediction result of the model, which is used to calculate the loss with the true label;

[0140] S53), use distillation loss To optimize:

[0141]

[0142] Where P teacher is the output of the teacher model; λ is a trade-off factor used to adjust the weight between the cross entropy loss and the MSE loss; MSE stands for Mean Squared Error Loss, which is used to calculate the difference between the student model prediction result and the teacher model output to help the student model better imitate the teacher model;

[0143] S54), the final loss of the cross-modal generator is:

[0144]

[0145] The probability model of the objective function is established through Bayesian optimization of each weighted coefficient, and the existing data points are used to optimize the weight distribution to adapt to different teacher-student structures. Finally, the location of myocardial infarction is completed through the classifier.

[0146] In addition, as shown in Table 1, this embodiment uses accuracy, specificity, sensitivity and F1 score for performance evaluation. The method of this embodiment achieves a recognition accuracy of 98.56% in cross-subject myocardial infarction recognition; the inter-patient test scheme on 6 types of MI localization tasks shows excellent performance, with an accuracy of 68.68%, which is significantly better than the existing myocardial infarction localization technology.

[0147] Table 1 Comparison of positioning results between the method of this embodiment and the existing method

[0148]

[0149]

[0150] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. A method for locating myocardial infarction based on cross-modal knowledge fusion, characterized in that: The steps include: S1), collecting target 1D-VCG signals and constructing a 2D-VCG image dataset, dividing the 2D-VCG image dataset into a training set and a test set; S2), constructing a teacher model and training it, extracting spatial features and pseudo-label Logits of 2D-VCG images through the teacher model; S3), constructing a student model to train it, and obtaining timing features and pseudo-label Logits from the 1D-VCG signal through the student model; S4), constructing a cross-modal generator, aligning and transforming the spatial features extracted by the teacher model and the temporal features extracted by the student model through the cross-modal generator; And cross-modal knowledge fusion and MI positioning are achieved through the cross-modal generator.

2. The method for locating myocardial infarction based on cross-modal knowledge fusion according to claim 1, characterized in that: In step S2), the teacher model is iteratively optimized based on the graph convolutional neural network MVCNN, and the teacher model obtains the features of each view by multiple parallel ResNet18 architectures; The core spatial features of each view are extracted and weighted aggregated through the multi-view collaborative attention fusion module.

3. The method for locating myocardial infarction based on cross-modal knowledge fusion according to claim 2, characterized in that: In step S2), the multi-view collaborative attention fusion module includes a two-stage attention mechanism, which enhances the semantic information of feature maps from different perspectives.

4. The method for locating myocardial infarction based on cross-modal knowledge fusion according to claim 3, characterized in that: In step S2), the relevant features of different perspectives are aggregated through the attention mechanism of the first stage, specifically: S211), the feature tensors (f i 1 , f i 2 , f i 3 ) compresses channel features through the convolution layer and combines them into a tensor The calculation expression is: f i Ang =ReLU(Conv(Cat(f i 1 ,f i 2 ,f i 3 ))) In the formula, (f i 1 , f i 2 , f i 3 ) are tensors of three different perspectives, and Cat represents the tensor views of three different perspectives (f i 1 , f i 2 , f i 3 ) are connected in series along the channel; Conv represents the convolution operation; ReLU is the activation function; the subscript i represents the index of the sample currently being processed, which is used to distinguish different samples; S212), calculating the attention factor through the sigmoid function; Right now: you A =σ(f i Ang ×1U A1 ×2U A2 ×3U A3 ) you B =σ(f i Ang ×1U B1 ×2U B2 ×3U B3 ) you C =σ(f i Ang ×1U C1 ×2U C2 ×3U C3 ) In the formula, u A ,u B ,u C is the attention factor, which indicates the feature weights under different perspectives; σ represents the sigmoid function; {U Ai , U Bi , U Ci } is the parameter matrix of the attention factor calculation module, which indicates the weight allocation to different feature channels; × indicates modular multiplication; A, B, and C indicate the labels of three different view feature channels respectively; S213), by multiplying the attention factor by the corresponding feature tensor element by element, an enhanced feature tensor is obtained Right now: In the formula, Represents the feature tensor of the first stage of enhanced attention; ⊙ represents element-by-element multiplication; Then two parallel internal attention feature fusion modules are used to enhance the semantic information of the feature map from different perspectives.

5. The method for locating myocardial infarction based on cross-modal knowledge fusion according to claim 4, characterized in that: In step S2), the second stage attention mechanism further interacts with the feature tensors from the first stage, and multiple multi-view feature tensors complement each other to form the final output; specifically, the following steps are included: S221), through the maximum pooling and average pooling operations, obtain and Right now: In the formula, Represents the feature tensor of the first stage of the enhanced attention mechanism, MaxPool and AvgPool represent the maximum pooling and average pooling operations respectively; Conv represents the convolution operation; ReLU is the activation function; and They represent the feature tensors after the maximum pooling MaxPool and average pooling AvgPool operations, respectively, to further extract the saliency information of the features; S222), combined with the self-attention mechanism, and the final feature tensor is generated through convolution operation Right now: In the formula, max means taking the average maximum value of elements along the channel dimension; Cat means concatenating two feature tensors along the channel dimension to integrate the feature information of different pooling operations; avg means taking the average value of elements along the channel dimension; Respectively represent the feature tensors The result after average pooling and maximum pooling operations; ⊙ represents element-by-element multiplication; σ represents the sigmoid function; Conv represents the convolution operation.

6. The method for locating myocardial infarction based on cross-modal knowledge fusion according to claim 1, characterized in that: In step S3), the student model adopts the ResNet18 architecture as a feature extractor and myocardial infarction localization classifier, and extracts the time-frequency features of the 1D-VCG timing signal by combining the cross-modal distillation technology. The maximum pooling layer and the fully connected layer of the student model are used to obtain the timing features and the output pseudo-labels, respectively, for subsequent knowledge enhancement optimization and student enhancement optimization in cross-modal knowledge distillation.

7. The method for locating myocardial infarction based on cross-modal knowledge fusion according to claim 1, characterized in that: In step S4), the cross-modal knowledge fusion includes knowledge enhancement optimization and student enhancement optimization.

8. The method for locating myocardial infarction based on cross-modal knowledge fusion according to claim 7, characterized in that: In step S4), the knowledge enhancement optimization refers to converting the spatial features of the cross-modal image obtained in the 2D-VCG into the Euclidean space, and independently transforming the features obtained by the teacher model and the student model into spatiotemporal features through the point generator, and performing loss supervision through the variational autoencoder to obtain the knowledge enhancement loss 9. The method for locating myocardial infarction based on cross-modal knowledge fusion according to claim 8, characterized in that: First, the 2D-VCG image features extracted by the teacher model are adjusted to fit the 1D-VCG temporal feature space of the student model; The cross-modal generator adopts a linear transformation strategy to ensure that the features from different modalities are accurately mapped and aligned in the Euclidean space; the cross-modal generator transforms the spatial features f of the 2D-VCG into 2D Converted to the timing feature f of 1D-VCG 1D Alignment form: f align =W·f 2D +b Where W is the conversion matrix; b is the bias term; f align is the transformed feature; The cross-modal generator uses a variational autoencoder (VAE) to further optimize features after feature alignment. The variational autoencoder (VAE) reconstructs a new feature representation consistent with the temporal feature space while ensuring the information richness of the features and adaptability to unseen data. The following objective function is optimized by minimizing the reconstruction error and KL divergence: in, represents the knowledge enhancement loss of the variational autoencoder (VAE); Encoder and Decoder represent the variational autoencoder VAE and decoder respectively, KL represents the Kullback-Leibler divergence, and β is a hyperparameter for adjusting the importance of KL divergence; f 1D Represents the timing characteristics of 1D-VCG; represents the standard normal distribution as the target for the latent space distribution.

10. The method for locating myocardial infarction based on cross-modal knowledge fusion according to claim 9, characterized in that: In step S5), the student enhancement optimization comprises the following steps: S51), input 1D-VCG into the encoder of the student model to generate a temporal feature representation f ts , and further optimized through a feature mapping layer U: f optimized =Uf ts In the formula, f optimized is the optimized feature of the feature mapping layer; U is the feature mapping layer; f ts Generate temporal feature representation; S52), the optimized feature f optimized Used to train the classifier and generate category prediction P pred ; and predict P by category pred Calculating cross entropy loss In the formula, CrossEntropy represents the cross entropy loss function, which is used to calculate the loss between the predicted result and the true label in the classification task, and y true Represents the true label, which is used to compare with the model prediction results. ;P pred Represents the category prediction result of the model, which is used to calculate the loss with the true label; S53), use distillation loss To optimize: Where P teacher is the output of the teacher model; λ is a trade-off factor used to adjust the weight between the cross entropy loss and the MSE loss; MSE stands for mean squared error loss, which is used to calculate the difference between the student model prediction results and the teacher model output to help the student model better imitate the teacher model; S54), the final loss of the cross-modal generator is: In the formula, α, β, and γ represent the weight coefficients of different loss terms, which are used to adjust the influence of each loss term in the final loss function; α corresponds to the knowledge enhancement loss The weight of β corresponds to the cross entropy loss The weight of γ corresponds to the distillation loss. The weight of Enhance loss for knowledge; is the cross entropy loss; is the distillation loss.

Images