Hierarchical multi-modal heart data completion method based on double knowledge guidance

Abstract

The invention provides a hierarchical multi-modal heart data completion method based on double knowledge guidance, which comprises the following steps: S1, extracting complete multi-modal joint features from complete multi-modal data through a teacher network, and carrying out cross-sample comparative learning and dynamic feature fusion on missing modal data through a student network to obtain a multi-modal data fusion model; generating completion features containing individual specific information; s2, based on the completion features, constructing a category prototype vector of the heart disease subtype, and generating semantic constraints representing disease subtype rules through prototype perception comparison distillation and moving average evolutionary strategies; s3, the complementation features and the semantic constraints are combined, network parameters and prototype vectors are optimized through back propagation iteration, complementation data are generated, and the complementation data comprise individual electrophysiological features and pathological subtype marks. By adopting the method, the individual specificity and the pathological evolution rule can be considered, and the reconstruction data can be ensured to meet the accurate heart disease diagnosis requirement at the same time.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent diagnosis of heart diseases, and in particular relates to a hierarchical multimodal heart data completion method based on dual knowledge guidance. Background Art

[0002] With the development of multimodal analysis technology in the field of intelligent diagnosis of cardiac diseases, cross-modal feature transfer and reconstruction technology has gradually become a research hotspot for solving the problem of missing modalities. This technology establishes a deep semantic mapping relationship between modalities and combines attention mechanisms, searchable feature maps, and contrastive autoencoders to attempt to extract complementary features from the remaining modalities and compensate for missing information. For example, it reconstructs the three-dimensional myocardial parameters of CMR images through the time-frequency features of ECG signals, or uses dynamic gating mechanisms to achieve adaptive fusion of multi-sequence image features, thereby improving the accuracy of pathological feature restoration.

[0003] Early studies mostly adopted a simplified strategy of directly removing missing modal samples. Although this reduced the complexity of the model, it led to further loss of already scarce sample data in the heart disease diagnosis scenario and was unable to cope with the dynamic missing problem in the testing phase. Although the methods based on cross-modal feature transfer alleviated the global semantic fault problem through mechanisms such as modal translation networks and context-aware compensation, such as assessing the degree of noise pollution through wavelet decomposition to directionally repair time series signals, or constructing contrastive autoencoders to achieve cross-modal phenotypic parameter restoration, these methods still focus on global feature alignment and rely on coarse-grained mapping of the latent space.

[0004] However, existing methods have three key flaws: First, during the reconstruction process, excessive attention is paid to the common laws between modalities and individual patient-specific information is ignored (such as differences in the time sensitivity of electrophysiological signals or the personalized distribution of myocardial parameters), resulting in the reconstructed data deviating from the true pathological state; second, there is a lack of explicit modeling of category-level pathological semantic constraints. For example, the discriminative features of myocardial ischemia and hypertrophic cardiomyopathy are not fully decoupled in the latent space, which can easily lead to misjudgment of disease subtypes; third, traditional frameworks have difficulty coordinating individual differences with disease evolution laws. The reconstruction results often reduce practical value due to insufficient anatomical structure rationality or weak clinical diagnostic relevance, which restricts the ability to accurately compensate in complex missing scenarios. Summary of the Invention

[0005] Based on this, it is necessary to provide a hierarchical multimodal cardiac data completion method based on dual knowledge guidance to address the above technical problems, which can take into account both individual specificity and pathological evolution laws, and ensure that the reconstructed data meets the needs of accurate heart disease diagnosis at the same time.

[0006] In a first aspect, the present application provides a hierarchical multimodal cardiac data completion method based on dual knowledge guidance, comprising:

[0007] S1. Extract complete multimodal joint features from complete multimodal data through the teacher network, and perform cross-sample comparative learning and dynamic feature fusion on the missing modality data through the student network to generate complementary features containing individual-specific information;

[0008] S2. Based on the completed features, we construct a category prototype vector for heart disease subtypes. Through prototype-aware contrast distillation and sliding average evolutionary strategies, we generate semantic constraints that characterize the regularities of disease subtypes.

[0009] S3. Combine the completed features with semantic constraints, iteratively optimize the network parameters and prototype vectors through back-propagation, and generate completed data. The completed data contains individual electrophysiological features and pathological subtype markers.

[0010] In a second aspect, the present application further provides a hierarchical multimodal cardiac data completion device based on dual knowledge guidance, comprising:

[0011] The cross-sample feature completion module is used to extract complete multimodal joint features from the complete multimodal data through the teacher network, and perform cross-sample comparative learning and dynamic feature fusion on the missing modality data through the student network to generate completed features containing individual-specific information;

[0012] A semantic evolution module for category-level prototype decoupling constraints, which is used to construct category prototype vectors for heart disease subtypes based on the completed features. This module generates semantic constraints that characterize the patterns of disease subtypes through prototype-aware contrast distillation and a sliding average evolutionary strategy.

[0013] The modality completion data generation module is used to jointly complete features and semantic constraints, iteratively optimize network parameters and prototype vectors through back-propagation, and generate completion data. The completion data contains individual electrophysiological characteristics and pathological subtype markers.

[0014] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned hierarchical multimodal cardiac data completion method based on dual knowledge guidance when executing the computer program.

[0015] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned hierarchical multimodal cardiac data completion method based on dual knowledge guidance.

[0016] This dual-knowledge-guided hierarchical multimodal cardiac data completion method utilizes a teacher network to extract complete multimodal joint features representing individual electrophysiological characteristics from complete multimodal data. Simultaneously, a student network applies cross-sample comparative learning to the missing modality data. Combined with a dynamic feature fusion strategy, it generates supplementary features containing individual-specific information, ensuring that the supplemented data aligns with the patient's own physiological characteristics. Based on these supplementary features, it constructs category prototype vectors for cardiac disease subtypes. Prototype-aware comparative distillation is used to extract common pathological patterns across disease subtypes. A sliding average evolutionary strategy dynamically updates semantic constraints to ensure that the supplemented data conforms to the evolutionary patterns of disease subtypes. Combining individual-specific supplementary features with disease subtype semantic constraints, the network parameters and prototype vectors are simultaneously optimized through backpropagation to generate supplementary data that contains both individual electrophysiological characteristics and pathological subtype labels. This approach, through a hierarchical supplementation mechanism, integrates cross-sample individual comparative knowledge with disease subtype semantic priors during the feature reconstruction phase. This addresses the diagnostic accuracy issues inherent in traditional methods, which often ignore individual patient differences or model pathological patterns in a simplistic manner. It effectively meets the dual requirements of data integrity and pathological consistency for accurate cardiac disease diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic flow chart of a hierarchical multimodal cardiac data completion method based on dual knowledge guidance provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a flow chart of a cross-sample feature completion method provided by an embodiment of the present invention;

[0020] Figure 3 A schematic structural diagram of a hierarchical multimodal cardiac data completion device based on dual knowledge guidance provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0022] First, a brief introduction is given to the terms involved in the embodiments of this application.

[0023] Prototype-aware contrastive distillation is a model training mechanism that integrates the common laws of disease subtypes with the alignment of individual features. Its core is to constrain the consistency of the completed features with the pathological semantics of the disease subtype by constructing a dynamically updated category prototype vector. Specifically, this method first aggregates the completed features of similar heart disease samples to generate a category prototype vector, which serves as a common feature template for this type of disease (such as the typical hemodynamic pattern of myocardial ischemia). Subsequently, by comparing the similarity matrices between the completed features and the prototype vectors calculated by the student network and the teacher network respectively (the student similarity matrix reflects the current distribution of the completed features, and the teacher similarity matrix provides a priori semantic constraints), a prototype distillation loss function is constructed to force the student network to retain individual specificity while aligning with the common pathological laws of the disease subtype when completing the missing modalities. This mechanism dynamically evolves the prototype vector through a sliding average strategy, allowing the semantic constraints to adapt to the temporal characteristics of disease evolution, ultimately ensuring that the completed data achieves a balance between individual differences and pathological rationality.

[0024] The sliding average evolutionary strategy is a method for dynamically updating category prototype vectors, aiming to enable the prototype vectors to adaptively fuse historical information with current data features, thereby stably characterizing the evolutionary patterns of disease subtypes. Specifically, the strategy introduces a momentum coefficient to fuse the prototype vector of the previous moment with the new features of the current batch of data in a weighted proportion, so that the update of the prototype vector retains historical semantic priors (such as the learned common features of myocardial hypertrophy) and gradually absorbs the supplementary information of new samples (such as the subtype variation patterns found in new cases). Through this smooth iterative mechanism, the prototype vector can avoid mutations caused by noise in a single batch of data, while continuously capturing the long-term evolutionary trends of disease subtypes (such as the stage-by-stage characteristic changes of chronic myocardial ischemia), ultimately ensuring that the completed data is both robust and timely in modeling pathological patterns.

[0025] Based on the above explanation of terms, the implementation environment of the hierarchical multimodal cardiac data completion method based on dual knowledge guidance provided in the embodiment of the present application is described. Schematically, the implementation environment includes: a terminal, a storage device, and a processor. The terminal is connected to the storage device and processor via a network or data bus; the storage device can be a distributed storage system or centralized storage; the processor includes but is not limited to a central processing unit, a multi-core processor, or an artificial intelligence chip, etc., which are not limited here.

[0026] In combination with the above-mentioned explanations of terms and implementation environment, the application scenarios of the embodiments of the present application are explained.

[0027] This technical solution is mainly used in cardiac disease diagnosis scenarios, especially when multimodal medical data (such as electrocardiograms, cardiac images, and medical records) are partially missing due to equipment failure, transmission errors, or privacy protection, it can effectively restore the missing information. For example, when a patient's cardiac magnetic resonance imaging data is lost or blurred, the system can use existing ECG signals and clinical texts, combined with cross-case pathological laws, to intelligently reconstruct missing imaging features (such as ventricular wall thickness and blood flow parameters). By integrating individual patient specificity with common knowledge of disease subtypes, the completed data significantly improves the robustness of the diagnostic model, allowing doctors to obtain reliable pathological judgments even when the data is incomplete. It is especially suitable for primary hospitals, remote consultations, or long-term health monitoring scenarios to assist in early disease screening and personalized treatment decisions.

[0028] Illustratively, the hierarchical multimodal cardiac data completion method based on dual knowledge guidance provided in the embodiment of the present application can also be applied to other application scenarios. It is only used as an example here and is not limited to the specific application scenario.

[0029] In an exemplary embodiment, Figure 1 As shown, a hierarchical multimodal cardiac data completion method based on dual knowledge guidance is provided. This embodiment uses the method as an example to illustrate the application of the method to a terminal in the aforementioned implementation environment. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps S1 to S3:

[0030] S1. The teacher network is used to extract complete multimodal joint features from the complete multimodal data, and the student network is used to perform cross-sample comparative learning and dynamic feature fusion on the missing modality data to generate complementary features containing individual-specific information.

[0031] Specifically, complete multimodal data including electrocardiogram signals, cardiac magnetic resonance images, and medical records can be obtained from the patient database of a medical institution. The complete multimodal data is input into a pre-trained teacher network, and multimodal joint features are extracted through a deep residual network (ResNet). Part of the modalities of the complete multimodal data are randomly masked (such as retaining only the electrocardiogram signal) to generate missing modal data and input into the student network. Through a cross-sample comparative learning mechanism, the reconstructed features generated by the student network are dynamically fused with the complete features of the teacher network to generate complete features containing individual patient-specific information (such as abnormal heart rhythm patterns and ventricular structural features). The advantage is that through the comparative supervision of the teacher-student architecture, it ensures that the complete features retain individual physiological characteristics while avoiding feature offsets caused by modality loss, thereby improving the generalization of subsequent diagnostic models.

[0032] S2. Based on the completed features, we construct the category prototype vectors of heart disease subtypes. Through prototype-aware contrast distillation and sliding average evolutionary strategies, we generate semantic constraints that characterize the laws of disease subtypes.

[0033] Specifically, based on the completed features generated in step S1, the features of disease samples of the same category are aggregated through the graph attention network (GAT) to construct the initial category prototype vector of the cardiac disease subtype (such as the common feature template of myocardial ischemia and myocardial hypertrophy); through prototype-aware contrast distillation, the similarity matrix between the completed features generated by the student network and the category prototype vector is calculated, and compared with the semantic prior provided by the teacher network to generate semantic constraints that constrain the pathological consistency of the completed features; further, a sliding average evolutionary strategy is adopted to fuse the prototype vectors of the historical batch data with the current batch features according to a preset momentum coefficient, and dynamically update the semantic constraints. Its advantage is that through dynamic prototype constraints, the completed data fits the real pathological evolution law of the disease subtype (such as the stage characteristics of chronic lesions), avoiding semantic deviations caused by static modeling.

[0034] S3. Combine the completed features with semantic constraints, iteratively optimize the network parameters and prototype vectors through back-propagation, and generate completed data. The completed data contains individual electrophysiological features and pathological subtype markers.

[0035] The completed features generated in step S1 are concatenated with the semantic constraints generated in step S2. The fused features are weighted using a channel attention mechanism to generate completed data that contains both individual electrophysiological features (such as QT interval parameters) and pathological subtype markers (such as abnormal myocardial ischemia velocity). Based on a backpropagation algorithm, the student network parameters and category prototype vectors are simultaneously optimized to achieve a balance between individual specificity and pathological consistency in the completed data. This approach has the advantage of ensuring, through an end-to-end joint training mechanism, that the completed data can be used for both individualized patient diagnosis (such as arrhythmia classification) and large-scale statistical analysis of disease subtypes (such as epidemiological studies of cardiomyopathy), thereby improving the availability and diagnostic reliability of medical data.

[0036] This dual-knowledge-guided hierarchical multimodal cardiac data completion method utilizes a teacher network to extract complete multimodal joint features representing individual electrophysiological characteristics from complete multimodal data. Simultaneously, a student network applies cross-sample comparative learning to the missing modality data. Combined with a dynamic feature fusion strategy, it generates supplementary features containing individual-specific information, ensuring that the supplemented data aligns with the patient's own physiological characteristics. Based on these supplementary features, it constructs category prototype vectors for cardiac disease subtypes. Prototype-aware comparative distillation is used to extract common pathological patterns across disease subtypes. A sliding average evolutionary strategy dynamically updates semantic constraints to ensure that the supplemented data conforms to the evolutionary patterns of disease subtypes. Combining individual-specific supplementary features with disease subtype semantic constraints, the network parameters and prototype vectors are simultaneously optimized through backpropagation to generate supplementary data that contains both individual electrophysiological characteristics and pathological subtype labels. This approach, through a hierarchical supplementation mechanism, integrates cross-sample individual comparative knowledge with disease subtype semantic priors during the feature reconstruction phase. This addresses the diagnostic accuracy issues inherent in traditional methods, which often ignore individual patient differences or model pathological patterns in a simplistic manner. It effectively meets the dual requirements of data integrity and pathological consistency for accurate cardiac disease diagnosis.

[0037] like Figure 2 As shown, in one possible embodiment, based on complete multimodal data and missing modal data, cross-sample comparative learning and dynamic feature fusion of the teacher network and the student network are used to generate complementary features containing individual-specific information, including:

[0038] S1.1. Input the complete multimodal data into the teacher network for joint feature embedding processing to generate complete multimodal joint features, where the complete multimodal data includes electrocardiogram signals, cardiac magnetic resonance images, and medical record text.

[0039] S1.2. Apply random modal masking to the complete multimodal data to generate missing modal data, and perform feature reconstruction on the missing modal data through the student network to obtain reconstructed features.

[0040] S1.3. Using the complete multimodal joint features as anchors, the reconstructed features of the same sample in the student network are taken to form positive sample pairs.

[0041] S1.4. Taking the complete multimodal joint features as anchor points, randomly select the complete multimodal joint features and reconstruction features of different samples to form the first negative sample pair and the second negative sample pair respectively. The first negative sample pair is used to characterize the missing reconstruction features of different samples in the teacher network and the student network, and the second negative sample pair is used to characterize the missing reconstruction features of different samples in the student network.

[0042] S1.5. Construct a cross-sample contrast loss function based on the complete multimodal joint features and the reconstructed features using the following formula:

[0043]

[0044] Where τ is the temperature coefficient, For complete multimodal joint features, To reconstruct features, I j≠i is the characteristic function, sim is the cosine similarity calculation function, is a positive sample pair, is the first negative sample pair, is the second negative sample pair.

[0045] Specifically, the cross-sample contrast loss function can effectively suppress the risk of feature space collapse through a two-way negative sampling strategy (considering negative samples between the teacher-student network and within the student network at the same time).

[0046] S1.6. Optimize the parameters of the student network based on the cross-sample contrast loss function to generate complementary features.

[0047] Specifically, the construction strategy of this technical method has two advantages: (1) positive sample pairs force the model to learn the invariance of the representation of individual patients in the complete-missing scenario, alleviating the feature shift caused by modality loss; (2) negative sample pairs enhance the model's ability to identify pathology specificity by explicitly distinguishing cross-patient heterogeneity. By constructing a contrastive learning mechanism with complete-missing data pairs from the same patient as positive sample pairs and heterogeneous data from different patients as negative sample pairs, the sample-level knowledge guidance module helps the student network effectively recover the semantic information of the missing modality by constraining the representation distance of the same sample in the teacher-student network to be minimized and the representation distance of different samples to be maximized.

[0048] In one possible embodiment, a cross-sample contrast loss function is constructed based on the complete multimodal joint features and the reconstructed features, including:

[0049] The feature similarities of the first negative sample pair, the second negative sample pair and the complete multimodal joint feature are calculated, and sorted based on the feature similarity to obtain a similarity sequence of all first negative sample pairs and second negative sample pairs.

[0050] Specifically, based on the complete multimodal joint features generated by the teacher network and the reconstructed features output by the student network, the cosine similarity of the first negative sample pair (teacher's complete features-other patients' student reconstructed features) and the second negative sample pair (student reconstructed features-other patients' student reconstructed features) is calculated; the similarities of all negative sample pairs are arranged in descending order to generate a similarity sequence. The implementation method may include: using batch data to parallelly calculate the similarity matrix between samples, or introducing a quick sorting algorithm to improve computational efficiency. Its advantage is that by quantifying feature similarity, it can identify potential confusing samples that are highly close to the anchor feature but actually belong to different patients, providing a data basis for subsequent screening of difficult negative samples.

[0051] The top-k difficult negative samples are screened according to the similarity order sequence. The difficult negative samples are negative sample pairs that have the highest similarity with the anchor point features and belong to different patients.

[0052] Specifically, by actively selecting highly similar heterogeneous samples, the model's ability to perceive subtle pathological differences (such as distinguishing early electrocardiogram features of myocardial ischemia from myocarditis) is enhanced, avoiding the problem of blurred judgment boundaries caused by an excessively high proportion of simple negative samples.

[0053] Based on the preset penalty strength coefficient, gradient penalty is applied to the contrast loss of difficult negative samples to optimize the discrimination boundary of the student network for heart disease subtypes.

[0054] For example, the domain applies a gradient penalty to the contrast loss of difficult negative sample pairs based on a preset penalty strength coefficient, and adjusts the student network parameters through backpropagation, forcing the model to increase the distance between difficult negative samples and anchor features in the feature space. Its advantage lies in improving the student network's accuracy in distinguishing heart disease subtypes (such as accurately distinguishing between stable and unstable angina imaging features) by targeted optimization of the distribution of highly confused samples, and enhancing the pathological interpretability of the supplemented features.

[0055] Specifically, this technical solution introduces a dynamic difficult sample mining mechanism. In each training batch, the top-k difficult negative samples (those with the highest similarity to the anchor points) are selected based on feature similarity, and a stronger gradient penalty is applied to them. This strategy can specifically optimize the decision boundary for easily confounded samples, improving the model's ability to distinguish heart disease subtypes in a fine-grained manner.

[0056] In one possible embodiment, the parameters of the student network are optimized based on a cross-sample contrast loss function to generate complementary features, including:

[0057] Based on the unmasked original modalities in the missing modal data, feature extraction is performed through the student network to obtain the currently available modal features, where the unmasked original modalities are the electrocardiogram signals, cardiac magnetic resonance images and medical records that have not been removed by random masking in the complete multimodal data.

[0058] Specifically, the unmasked original modalities in the missing modal data (such as the retained electrocardiogram signal or cardiac magnetic resonance image) are processed through the feature extraction module of the student network to generate the currently available modal features. For example, a pre-trained deep neural network (such as ResNet-50) can be used to extract spatial features of the image data, or a bidirectional long short-term memory network (BiLSTM) can be used to perform time series modeling on the electrocardiogram signal. The advantage of this step is that it fully retains the original physiological information of the unmasked modality (such as the ST segment offset feature of the electrocardiogram signal), provides a reliable individual observation benchmark for subsequent fusion, and avoids the loss of feature information due to modality loss.

[0059] Based on the cross-sample contrast loss function, the reconstructed features of the student network are subjected to cross-sample correlation constraints to obtain cross-sample correlation features that characterize the similarities and differences between samples.

[0060] Specifically, based on the cross-sample contrast loss function, cross-sample correlation constraints are imposed on the reconstructed features of the student network. By comparing the differences in feature distribution between the same patient and different patients, correlation features that characterize the similarities between samples and the differences in disease subtypes are generated. For example, the strength of the association between samples can be quantified by constructing a feature similarity matrix (such as a cosine similarity matrix), or a graph convolutional network can be introduced to model the pathological association topology between patients. The advantage of this step is that through feature association learning at the group level, the ability of the complementary features to capture the common laws of disease subtypes (such as the typical hemodynamic pattern of myocardial ischemia) is enhanced, thereby improving the generalization of features in cross-patient diagnosis.

[0061] The following formula is used to adaptively weight the currently available modal features and cross-sample correlation features to generate the completed features:

[0062] M fused =α·M obs +(1-α)·M distill

[0063] Among them, M fused To complete the feature, M obs is the currently available modal feature, M distill is the cross-sample correlation feature, and α is a dynamically adjusted learnable parameter.

[0064] Specifically, the currently available modal features and cross-sample correlation features are input into the adaptive fusion module, and the fusion weights of the two types of features are adjusted through dynamic learnable parameters to generate complementary features. For example, a gating network based on the attention mechanism can be designed to dynamically adjust the weight distribution according to the confidence level of the input modality (such as the electrocardiogram signal quality score), or a multi-layer perceptron can be used to predict the fusion coefficient. The above technical method significantly improves the robustness of feature compensation under complex missing patterns by explicitly modeling the prior constraints on similarity between samples.

[0065] In one possible embodiment, based on the completed features, a category prototype vector of heart disease subtypes is constructed. Through prototype-aware contrast distillation and sliding average evolutionary strategies, semantic constraints representing the regularity of disease subtypes are generated, including:

[0066] Based on the completed features, the graph attention network aggregates the features of samples of the same category to generate the initial category prototype vector using the following formula:

[0067]

[0068] Among them, c k is the initial category prototype vector, A k is the sample set of category k in the current batch of samples, P i Indicates A k The i-th sample in M i A k The multimodal features of the i-th sample in .

[0069] Specifically, in the study of modality missing completion guided by category-level prototypes, constructing a cardiac disease category prototype is one of the key links. The completion features belonging to the same cardiac disease subtype (such as myocardial ischemia or myocardial hypertrophy) are screened from the samples of the current training batch, and the feature association weights between similar samples are calculated through the graph attention network. The completion features are weighted aggregated to generate the initial category prototype vector. For example, a clustering algorithm (such as K-means) can be used to group similar samples, or the subtype category set can be directly divided based on the disease label library. The above technical methods can effectively solve the problem of chaotic distribution between categories. Even if there is a modality missing, the model can use the modality completion mechanism to accurately classify patients into the correct cardiac disease category.

[0070] The similarity between the initial category prototype vector and the completed feature and the similarity between the initial category prototype vector and the completed feature are calculated using the following formulas to generate the student similarity matrix and the teacher similarity matrix:

[0071]

[0072] w∈{t,s}

[0073] Among them, F k(i) To characterize the sample P i with c k The similarity between is the student similarity matrix, is the teacher similarity matrix.

[0074] Specifically, the cosine similarity between the complementary features generated by the student network and the initial category prototype vector is calculated to generate a student similarity matrix; the similarity between the complete multimodal features provided by the teacher network and the prototype vector is simultaneously calculated to generate a teacher similarity matrix. For example, similarity can be calculated in parallel using batch samples, or normalization processing can be introduced to eliminate feature scale differences. The advantage of this step is that it quantifies the semantic association strength between the complementary features and the disease subtype prototype (such as the matching degree of a complementary feature with the myocardial hypertrophy prototype), providing an optimizable comparison target for the distillation loss.

[0075] According to the following prototype distillation loss function, the category prototype distillation loss is calculated based on the teacher similarity matrix and the student network similarity matrix, and the student network is optimized by minimizing the category prototype distillation loss:

[0076]

[0077] Among them, L C is the category prototype distillation loss, N is the total number of samples, and K is the category identifier of the batch samples.

[0078] Specifically, the similarity matrix of the sample and the category prototype is calculated by the teacher network and the student network respectively. and Then, by minimizing the Euclidean distance between the two similarity matrices, the student model can learn similar inter-category and intra-category feature variations as the teacher model, thereby better capturing the consistency and difference feature representations between different categories and effectively solving the problem of chaotic distribution between categories.

[0079] In one possible embodiment, based on the completed features, a category prototype vector of heart disease subtypes is constructed, and semantic constraints representing the regularity of disease subtypes are generated through prototype-aware contrast distillation and sliding average evolutionary strategies. The following also includes:

[0080] According to the following formula, the reconstructed features and the initial category prototype vector are iterated through the sliding average strategy to generate a dynamic category prototype vector:

[0081]

[0082] in, is the dynamic category prototype vector, β is the momentum coefficient, To reconstruct features.

[0083] Based on the category prototype vectors accumulated during the historical training process and the reconstructed features generated by the student network of the current batch, a sliding average strategy is used for progressive fusion to generate a dynamically evolving category prototype vector. For example, the momentum coefficient can be set to control the retention ratio of the historical prototype vector and the contribution of the current feature update. For example, the weight of historical information is set to 0.9 and the weight of the current batch feature is set to 0.1 to achieve smooth iteration of the prototype vector. The advantage of this step is that the category prototype can continuously absorb the fine-grained pathological features of new cases (such as changes in blood flow parameters in different stages of myocardial ischemia) while retaining the common laws of diseases learned over a long period of time (such as typical imaging signs of myocardial hypertrophy), thereby adapting to the dynamic evolution of disease representation.

[0084] Use the dynamic category prototype vector as pathological semantic prior knowledge and re-execute S1.6 to optimize the completion features.

[0085] Specifically, the dynamically updated category prototype vector is used as pathological semantic prior knowledge and re-involved in the cross-sample contrast loss calculation. The student network parameters are optimized through backpropagation to force the completion features to align with the disease subtype patterns represented by the dynamic prototype. For example, the semantic similarity between the completion features and the dynamic prototype can be constrained by a contrastive learning mechanism, or a distillation loss function can be designed to enforce feature distribution consistency. The advantage of this step is that it uses the latest pathological patterns encoded in the dynamic prototype (such as the myocarditis subtype features in the new international classification standard) to guide the completion features to accurately match the diagnostic criteria of the disease subtype, avoiding feature degradation caused by historical data bias. The above technical method realizes the differentiable dynamic evolution of the category prototype through a sliding average strategy, enabling the completion features to adapt to the fine-grained evolution of the disease subtype (such as the stage-specific pathological features of chronic myocardial ischemia). As a continuously updated semantic prior, the dynamic prototype vector effectively balances historical experience and new case knowledge, significantly improving the ability of the cardiac disease diagnosis model to capture complex pathological changes (such as the identification of early asymptomatic myocardial ischemia), while enhancing the diagnostic consistency of the completion data in cross-time and cross-device scenarios.

[0086] In one possible embodiment, the completion features and semantic constraints are combined to iteratively optimize network parameters and prototype vectors through back-propagation to generate completion data, including:

[0087] Based on the cross-sample contrast loss function and the prototype distillation loss function, the total loss function is generated by weighted summation.

[0088] For example, preset weight coefficients can be assigned to the two loss functions, such as setting the weight of the cross-sample comparison loss to 1.0 and the weight of the prototype distillation loss to 0.5, to balance the optimization goals of individual feature restoration and alignment with pathological laws. The advantage of this step is that by adjusting the loss weights, the model can take into account both individual patient specificity (such as ECG signal details) and common characteristics of disease subtypes (such as typical imaging patterns of myocardial ischemia) during training, avoiding feature bias caused by a single loss dominance.

[0089] Based on the total loss function, the parameters of the student network and the initial category prototype vector are synchronously updated and iterated through the gradient descent algorithm to obtain the optimized parameters of the student network and the optimized category prototype vector.

[0090] Specifically, based on the total loss function, a gradient descent algorithm is used to synchronously update and iterate the parameters of the student network and the initial category prototype vector. For example, an adaptive optimizer is used to dynamically adjust the learning rate to ensure that the network parameters and the prototype vector are collaboratively optimized during the backpropagation process. The advantage of this step is that through the joint optimization mechanism, the reconstruction ability of the student network and the semantic representation of the prototype vector promote each other. For example, while improving the accuracy of the individual completed features, it enhances its adaptability to the latest disease diagnostic standards (such as the myocarditis classification guidelines).

[0091] Based on the optimized student network optimization parameters and optimized category prototype vector, the completion features are generated.

[0092] Specifically, the optimized student network parameters and category prototype vectors are loaded into the feature generation module to complete the feature reconstruction of the missing modal data. For example, an end-to-end feature decoder can be designed to map the fused feature vector to the target modal space (such as the pixel space of cardiac magnetic resonance imaging). The advantage of this step is that the generated completed data retains the patient's physiological details (such as ventricular wall thickness) and disease subtype markers (such as blood flow parameters of myocardial hypertrophy), providing high-fidelity multimodal data support for clinical diagnosis. The above technical method achieves the optimal balance between individual restoration accuracy and pathological semantic consistency of the completed features through joint loss optimization and synchronous parameter update, significantly improving the robustness of the heart disease diagnosis model in complex missing scenarios. For example, when both image and text data are missing, complete features that conform to the laws of disease subtypes can still be reconstructed based on electrocardiogram signals to assist doctors in making accurate diagnoses.

[0093] In summary, the hierarchical multimodal cardiac data completion method based on dual knowledge guidance provided by the embodiment of the present application, the teacher network extracts complete joint features from complete multimodal data (including electrocardiogram signals, cardiac images and medical records) as a benchmark for individual-specific knowledge; the student network performs cross-sample comparative learning on the missing data after random masking, and reconstructs the distribution consistency of features and complete features through positive and negative sample pairs, while fusing the unmasked original modal features and cross-sample correlation features to generate completed features that retain the patient's unique physiological characteristics; based on the completed features, a prototype vector of the disease subtype category is constructed, and the similarity matrix difference between the student network and the teacher network is calculated through prototype-aware comparative distillation to generate semantic constraints that characterize the common pathological laws, and a sliding average strategy is used to dynamically iterate the prototype vector to make it adaptive to the fine-grained evolution of disease subtypes; the completed features and dynamic semantic constraints are combined, and the network parameters and prototype vectors are synchronously optimized through a weighted loss function to generate completed data that simultaneously embeds individual electrophysiological features and pathological subtype labels. The above-mentioned technical solution uses a hierarchical completion mechanism to integrate cross-sample individual comparison knowledge and dynamic prototype semantic priors in the feature reconstruction stage, solving the completion distortion problem caused by traditional methods that ignore individual differences among patients or static modeling pathological laws. It can take into account both individual specificity and pathological evolution laws, ensuring that the reconstructed data retains the patient's unique physiological details such as ventricular structure and rhythm pattern while strictly complying with the diagnostic standards and evolution laws of disease subtypes such as myocardial ischemia and myocardial hypertrophy, thereby improving the accuracy and robustness of heart disease diagnostic models in data-missing scenarios, and providing reliable multimodal data support for clinical precision diagnosis.

[0094] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0095] Based on the same inventive concept, embodiments of the present application further provide a dual-knowledge-guided hierarchical multimodal cardiac data completion device 10 for implementing the aforementioned dual-knowledge-guided hierarchical multimodal cardiac data completion method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the dual-knowledge-guided hierarchical multimodal cardiac data completion device 10 provided below can be found in the above-mentioned limitations of the dual-knowledge-guided hierarchical multimodal cardiac data completion method, and will not be repeated here.

[0096] In an exemplary embodiment, Figure 3 As shown, a hierarchical multimodal cardiac data completion device 10 based on dual knowledge guidance is provided, comprising:

[0097] The cross-sample feature completion module 11 is used to extract complete multimodal joint features from the complete multimodal data through the teacher network, and perform cross-sample comparative learning and dynamic feature fusion on the missing modality data through the student network to generate completed features containing individual-specific information.

[0098] The semantic evolution module 12 of category-level prototype decoupling constraints is used to construct category prototype vectors of heart disease subtypes based on the completed features, and to generate semantic constraints that characterize the laws of disease subtypes through prototype-aware contrast distillation and sliding average evolutionary strategies.

[0099] The modality completion data generation module 13 is used to combine the completion features and semantic constraints, iteratively optimize the network parameters and prototype vectors through back propagation, and generate the completion data. The completion data includes individual electrophysiological characteristics and pathological subtype markers.

[0100] In a possible embodiment, the cross-sample feature completion module 11 includes:

[0101] The multimodal feature embedding unit is used to input the complete multimodal data into the teacher network for joint feature embedding processing to generate complete multimodal joint features, where the complete multimodal data includes electrocardiogram signals, cardiac magnetic resonance images and medical record texts.

[0102] The missing modality generation unit is used to apply random modality masking to the complete multimodal data to generate missing modality data, and perform feature reconstruction processing on the missing modality data through the student network to obtain reconstructed features.

[0103] The positive sample pair construction unit is used to take the complete multimodal joint features as anchors and take the reconstructed features of the same sample in the student network to form a positive sample pair.

[0104] The negative sample construction unit is used to randomly select the complete multimodal joint features and reconstruction features of different samples with the complete multimodal joint features as anchor points to form the first negative sample pair and the second negative sample pair respectively. The first negative sample pair is used to characterize the missing reconstruction features of different samples in the teacher network and the student network, and the second negative sample pair is used to characterize the missing reconstruction features of different samples in the student network.

[0105] The loss function construction unit is used to construct a cross-sample comparison loss function based on the complete multimodal joint features and the reconstructed features using the following formula:

[0106]

[0107] Where τ is the temperature coefficient, For complete multimodal joint features, To reconstruct features, I j≠i is the characteristic function, sim is the cosine similarity calculation function, is a positive sample pair, is the first negative sample pair, is the second negative sample pair.

[0108] The parameter optimization unit is used to optimize the parameters of the student network based on the cross-sample contrast loss function to generate complementary features.

[0109] In a possible embodiment, the loss function construction unit includes:

[0110] The similarity sorting unit is used to calculate the feature similarity of the first negative sample pair, the second negative sample pair and the complete multimodal joint feature, and sort them based on the feature similarity to obtain a similarity sequence of all first negative sample pairs and second negative sample pairs.

[0111] The difficult negative sample screening unit is used to screen the top-k difficult negative samples according to the similarity order sequence. The difficult negative samples are negative sample pairs that have the highest similarity with the anchor point features and belong to different patients.

[0112] The boundary optimization unit is used to perform gradient penalty processing on the contrast loss of difficult negative samples based on a preset penalty strength coefficient to optimize the student network's discrimination boundary for heart disease subtypes.

[0113] In a possible embodiment, the parameter optimization unit includes:

[0114] The observation feature extraction unit is used to extract features based on the unmasked original modalities in the missing modal data through the student network to obtain the currently available modal features, where the unmasked original modalities are the electrocardiogram signals, cardiac magnetic resonance images and medical record texts in the complete multimodal data that have not been removed by random masking processing.

[0115] The correlation feature generation unit is used to perform cross-sample correlation constraints on the reconstructed features of the student network based on the cross-sample contrast loss function, and obtain cross-sample correlation features that characterize the similarities and differences between samples.

[0116] The adaptive fusion unit is used to perform adaptive weight fusion of the currently available modal features and cross-sample correlation features using the following formula to generate complementary features:

[0117] M fused =α·M obs +(1-α)·M distill

[0118] Among them, M fused To complete the feature, M obs is the currently available modal feature, M distill is the cross-sample correlation feature, and α is a dynamically adjusted learnable parameter.

[0119] In a possible embodiment, the semantic evolution module 12 of the category-level prototype decoupling constraint includes:

[0120] The prototype generation unit is used to generate the initial category prototype vector by aggregating the features of samples of the same category through the graph attention network based on the completed features using the following formula:

[0121]

[0122] Among them, c k is the initial category prototype vector, A k is the sample set of category k in the current batch of samples, P i Indicates A k The i-th sample in M i A k The multimodal features of the i-th sample in .

[0123] The similarity matrix construction unit is used to calculate the similarity between the initial category prototype vector and the completed feature and the similarity between the initial category prototype vector and the completed feature using the following formula to generate the student similarity matrix and the teacher similarity matrix:

[0124]

[0125] w∈{t,s}

[0126] Among them, F k(i) To characterize the sample P i with c k The similarity between is the student similarity matrix, is the teacher similarity matrix.

[0127] The distillation loss optimization unit is used to calculate the category prototype distillation loss based on the teacher similarity matrix and the student network similarity matrix according to the following prototype distillation loss function, and optimize the student network by minimizing the category prototype distillation loss:

[0128]

[0129] Among them, L C is the category prototype distillation loss, N is the total number of samples, and K is the category identifier of the batch samples.

[0130] In a possible embodiment, the semantic evolution module 12 of the category-level prototype decoupling constraint further includes:

[0131] The iterative update unit is used to iterate the reconstructed features and the initial category prototype vector using a sliding average strategy according to the following formula to generate a dynamic category prototype vector:

[0132]

[0133] in, is the dynamic category prototype vector, β is the momentum coefficient, To reconstruct features.

[0134] The prior optimization unit is used to use the dynamic category prototype vector as pathological semantic prior knowledge, re-call the parameter optimization unit, and optimize the completion features.

[0135] In a possible embodiment, the modality completion data generation module 13 includes:

[0136] The total loss construction unit is used to generate the total loss function through weighted summation based on the cross-sample contrast loss function and the prototype distillation loss function.

[0137] The synchronous update unit is used to synchronously update and iterate the parameters of the student network and the initial category prototype vector based on the total loss function through the gradient descent algorithm to obtain the optimized parameters of the student network and the optimized category prototype vector.

[0138] The feature generation unit is used to generate completion features based on the optimized student network optimization parameters and the optimized category prototype vector.

[0139] In one embodiment, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the hierarchical multimodal cardiac data completion method based on dual knowledge guidance as described above are implemented.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0141] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0142] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A hierarchical multimodal cardiac data completion method based on dual knowledge guidance, characterized in that: The method comprises: S1. Extract complete multimodal joint features from complete multimodal data through the teacher network, and perform cross-sample comparative learning and dynamic feature fusion on the missing modality data through the student network to generate complementary features containing individual-specific information; S2. Based on the completed features, construct a category prototype vector of the heart disease subtype, and generate semantic constraints that characterize the regularity of the disease subtype through prototype-aware contrast distillation and sliding average evolutionary strategy; S3. Combining the completion features with the semantic constraints, iteratively optimizing network parameters and prototype vectors through back propagation to generate completion data, where the completion data includes individual electrophysiological features and pathological subtype markers.

2. The method according to claim 1, characterized in that The teacher network is used to extract complete multimodal joint features from the complete multimodal data, and the student network is used to perform cross-sample comparative learning and dynamic feature fusion on the missing modality data to generate complementary features containing individual-specific information, including: S1.

1. Inputting the complete multimodal data into the teacher network for joint feature embedding processing to generate a complete multimodal joint feature, wherein the complete multimodal data includes an electrocardiogram signal, a cardiac magnetic resonance image, and a medical record text; S1.

2. applying random modality masking to the complete multimodal data to generate missing modality data, and performing feature reconstruction on the missing modality data using the student network to obtain reconstructed features; S1.

3. Using the complete multimodal joint feature as an anchor, take the reconstructed features of the same sample in the student network to form a positive sample pair; S1.

4. Using the complete multimodal joint feature as an anchor point, randomly select the complete multimodal joint feature and the reconstruction feature of different samples to form a first negative sample pair and a second negative sample pair, respectively, wherein the first negative sample pair is used to characterize the missing reconstruction features of different samples in the teacher network and the student network, and the second negative sample pair is used to characterize the missing reconstruction features of different samples in the student network; S1.

5. Construct a cross-sample contrast loss function based on the complete multimodal joint features and the reconstructed features using the following formula: Where τ is the temperature coefficient, is the complete multimodal joint feature, is the reconstruction feature, I j≠i is the characteristic function, sim is the cosine similarity calculation function, is the positive sample pair, is the first negative sample pair, is the second negative sample pair; S1.

6. Optimize the parameters of the student network based on the cross-sample contrast loss function to generate the complementary features.

3. The method according to claim 2, characterized in that The constructing a cross-sample contrast loss function based on the complete multimodal joint feature and the reconstructed feature includes: Calculating feature similarities between the first negative sample pair, the second negative sample pair, and the complete multimodal joint feature, and sorting them based on the feature similarities to obtain a similarity sequence of all first negative sample pairs and second negative sample pairs; Filter the top-k difficult negative samples according to the similarity order sequence, where the difficult negative samples are negative sample pairs that have the highest similarity with the anchor point features and belong to different patients; Based on a preset penalty intensity coefficient, gradient penalty processing is performed on the contrast loss of the difficult negative sample to optimize the discrimination boundary of the student network for the heart disease subtype.

4. The method according to claim 2, characterized in that The optimizing the parameters of the student network based on the cross-sample contrast loss function to generate the complementary features includes: Based on the unmasked original modalities in the missing modality data, feature extraction is performed through the student network to obtain currently available modality features, wherein the unmasked original modalities are the electrocardiogram signals, cardiac magnetic resonance images, and medical record texts in the complete multimodal data that have not been removed by random masking; Performing cross-sample correlation constraints on the reconstructed features of the student network based on the cross-sample contrast loss function to obtain cross-sample correlation features that characterize the similarities and differences between samples; The current available modality feature and the cross-sample correlation feature are adaptively weighted and fused using the following formula to generate the complementary feature: M fused =α·M obs +(1-a)·M distill Among them, M fused is the completion feature, M obs is the currently available modal feature, M distill is the cross-sample correlation feature, and α is a dynamically adjusted learnable parameter.

5. The method according to claim 2, characterized in that Based on the completed features, a category prototype vector of the heart disease subtype is constructed, and semantic constraints representing the regularity of the disease subtype are generated through prototype-aware contrast distillation and sliding average evolutionary strategies, including: Based on the completed features, the graph attention network aggregates the features of samples of the same category to generate the initial category prototype vector using the following formula: Among them, c k is the initial category prototype vector, A k is the sample set of category k in the current batch of samples, P i Indicates A k The i-th sample in M i A k The multimodal features of the i-th sample in ; The similarity between the initial category prototype vector and the supplementary feature and the similarity between the initial category prototype vector and the supplementary feature are calculated respectively by the following formula to generate a student similarity matrix and a teacher similarity matrix: Among them, F k(i) To characterize the sample P i with c k The similarity between is the student similarity matrix, is the teacher similarity matrix; According to the following prototype distillation loss function, the category prototype distillation loss is calculated based on the teacher similarity matrix and the student network similarity matrix, and the student network is optimized by minimizing the category prototype distillation loss: Among them, L C is the category prototype distillation loss, N is the total number of samples, and K is the category identifier of the batch samples.

6. The method according to claim 5, characterized in that The method further includes: constructing a category prototype vector of the heart disease subtype based on the completed features, and generating semantic constraints representing the regularity of the disease subtype through prototype-aware contrast distillation and sliding average evolutionary strategy. According to the following formula, the reconstructed features and the initial category prototype vector are iterated by a sliding average strategy to generate a dynamic category prototype vector: in, is the prototype vector of the dynamic category, β is the momentum coefficient, is the reconstructed feature; The dynamic category prototype vector is used as pathological semantic prior knowledge, and S1.6 is re-executed to optimize the completion feature.

7. The method according to claim 5, characterized in that The combining of the completion features and the semantic constraints, iteratively optimizing network parameters and prototype vectors through back-propagation, and generating completion data includes: Based on the cross-sample contrast loss function and the prototype distillation loss function, generating a total loss function through weighted summation; Based on the total loss function, the parameters of the student network and the initial category prototype vector are synchronously updated and iterated by the gradient descent algorithm to obtain the optimized parameters of the student network and the optimized category prototype vector; The completion feature is generated based on the optimized student network optimization parameters and the optimized category prototype vector.

8. A hierarchical multimodal cardiac data completion device based on dual knowledge guidance, characterized in that: The device comprises: The cross-sample feature completion module is used to extract complete multimodal joint features from the complete multimodal data through the teacher network, and perform cross-sample comparative learning and dynamic feature fusion on the missing modality data through the student network to generate completed features containing individual-specific information; A semantic evolution module for category-level prototype decoupling constraints, which is used to construct category prototype vectors of heart disease subtypes based on the completed features, and generate semantic constraints that characterize the laws of disease subtypes through prototype-aware contrast distillation and sliding average evolutionary strategies; The modality completion data generation module is used to combine the completion features with the semantic constraints, iteratively optimize network parameters and prototype vectors through back propagation, and generate completion data, wherein the completion data includes individual electrophysiological characteristics and pathological subtype markers.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images