A method for feature alignment and co-encoding of multi-modal electrocardiogram data
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG WANLI UNIV
- Filing Date
- 2026-02-10
- Publication Date
- 2026-06-05
Smart Images
Figure CN122156329A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of electrocardiogram (ECG) data processing, specifically a feature alignment and collaborative coding method for multimodal ECG data. Background Technology
[0002] Multimodal electrocardiogram (ECG) data refers to datasets that combine ECG signals with other relevant clinical or auxiliary information, aiming to improve the accuracy and robustness of disease diagnosis by fusing information from different modalities. Feature alignment refers to mapping data from different modalities to the same semantic space, making cross-modal features comparable and correlated in mathematical representation, thereby addressing the problem of intermodal heterogeneity (such as the time-series data of ECG versus the discrete symbols of text). Cooperative coding refers to dynamically integrating multimodal features through interactive mechanisms (such as attention mechanisms and gating fusion) to generate richer joint representations to capture complex intermodal relationships (such as the causal relationship between ECG abnormalities and symptoms). Multimodal ECG data improves diagnostic capabilities by fusing information from different modalities, but the problems of modal heterogeneity and semantic gaps need to be addressed. Feature alignment is the foundation of cross-modal understanding, establishing semantic correspondences between modalities through contrastive learning, projection, and other methods. Co-coding dynamically integrates multimodal features through mechanisms such as attention and gating to generate more robust joint representations. The combination of these two approaches can significantly improve the performance of tasks such as cardiovascular disease diagnosis and telemedicine, and is currently a hot research direction in multimodal medical AI.
[0003] Existing ECG data processing solutions mostly focus on single-dimensional alignment, neglecting semantic association and physiological significance. They fail to enhance key semantics related to cardiac health in bimodal features through attention mechanisms, failing to highlight core features and weaken redundant features. The lack of physiological association constraints results in weak correlation between aligned features and cardiac health status, or even alignment results without physiological meaning, deviating from clinical diagnostic needs and failing to provide a high-quality foundation for subsequent encoding. Furthermore, existing modality fusion technologies often employ fixed-weight strategies without introducing adaptive modality weight learning, failing to dynamically adjust the contribution of time-series and image modalities based on individual physiological characteristics. In complex scenarios, they cannot flexibly adapt to differences in feature reliability across different modalities, easily leading to problems such as "insufficient contribution from dominant modalities and excessive interference from inferior modalities," resulting in rigid fusion effects and poor adaptability. Existing technologies often use single-granularity annotation systems, with coarse annotation methods that cannot meticulously and comprehensively characterize the association between bimodal features and cardiac health status. This results in low effectiveness of the labeled data, insufficient supervision information for model training, and consequently, low model learning accuracy, failing to accurately uncover the correlation between features and diseases. Summary of the Invention
[0004] This invention aims to at least solve one of the technical problems existing in the prior art; to this end, this invention proposes a feature alignment and collaborative coding method for multimodal electrocardiogram data, which solves the technical problem that most methods only focus on alignment in a single dimension, ignore semantic association and physiological meaning, fail to strengthen the key semantics related to heart health in bimodal features through attention mechanisms, and fail to highlight the core features.
[0005] To address the aforementioned problems, a first aspect of the present invention provides a method for feature alignment and collaborative coding of multimodal electrocardiogram data, comprising the following steps: An ECG time series was constructed, and the ECG time series signal was converted into a 12-lead standard image. The 12 channels correspond to the 12 leads, and the spatial correlation features between leads were preserved. In order to meet the requirements of ECG-Thinking multimodal processing, a dual-modal dataset of ECG time series signal combined with 12-lead ECG image was constructed, and targeted noise suppression was performed and modality adaptation was completed. We extract bimodal features from bimodal datasets and divide bimodal feature alignment into distribution alignment stage, attention semantic alignment stage, and physiological association constraint stage. We generate a stage strategy to map bimodal features to a unified feature space. Based on the dual-modal features mapped to a unified feature space, a comprehensive strategy of adaptive modality fusion and multi-scale convolutional coding is established to perform deep co-coding of the dual-modal features and output the final multi-scale co-coded features. Based on multi-scale co-coding features, a hierarchical multi-granularity labeling system is used to label the bimodal dataset. Based on the labeled bimodal dataset, a SFT-supervised fine-tuning deep learning model is trained, and the model with the highest accuracy is selected as the optimal model in the SFT stage. The multi-scale co-coding features are then analyzed. Based on the optimal model in the SFT stage, a data flywheel mechanism is introduced to perform data flywheel iteration and optimize the optimal model in the SFT stage.
[0006] Optionally, in one example of the above aspects, constructing an ECG time series and converting the ECG time series signal into a 12-lead standard image, with 12 channels corresponding to 12 leads, while preserving the spatial correlation features between leads, includes the following steps: Construct an ECG time series, including core and auxiliary signals; Set the size and resolution of the generated 12-lead standard image, construct a two-dimensional coordinate system for each lead's ECG signal, with the horizontal axis representing time and the vertical axis representing voltage, and discretize the preprocessed ECG signal into a series of points in chronological order, with each point corresponding to a time-voltage coordinate; The 12-lead ECG is encoded based on the spatial coordinates between the leads. The 12-lead ECG includes standard limb leads and chest leads. The 12 channels in the standard 12-lead image are set to correspond to the signals of the 12 leads. The channels corresponding to the standard limb leads are arranged in the upper half of the image, and the channels corresponding to the chest leads are arranged in the lower half of the image. In the coordinate system corresponding to each lead, the waveform of the ECG signal is plotted according to the discretized time-voltage coordinates. The lead code is added next to the waveform of each lead as an identifier to obtain the 12-lead standard image of the ECG time series signal conversion.
[0007] Optionally, in one example of the above aspects, to meet the requirements of ECG-Thinking multimodal processing, a dual-modal dataset of ECG time-series signals combined with 12-lead ECG images is constructed, targeted noise suppression is performed, and modality adaptation is completed, including the following steps: Median filtering is applied to the ECG time series signal, adaptive notch filtering is applied to the median-filtered ECG time series signal, and wavelet threshold denoising is applied to the filtered signal to suppress various types of noise in steps. The respiratory signal was filtered by moving average, the body movement signal was filtered by low-pass, and the blood oxygen signal was filtered by exponential smoothing to suppress auxiliary signal noise. Gaussian filtering was used to denoise the 12-lead standard image. The ECG time series signal was standardized by Z-score standardization, and the converted 12-lead standard image values were normalized. At the same time, the image size was adjusted to a uniform specification. By using a timestamp alignment algorithm, the time dimension of the standardized ECG time series signal is made consistent with that of the normalized 12-lead standard image, thus completing modality adaptation.
[0008] Optionally, in one example of the above aspects, extracting bimodal features from a bimodal dataset includes the following steps: Extract nonlinear and linear features to construct a time series feature set; for nonlinear features, extract the maximum Lyapunov exponent, correlation dimension, and sample entropy; for linear features, extract the mean / standard deviation of RR intervals and QRS amplitude / duration to extract time series modal features. Spatial feature data of 12-lead standard images were extracted using a lightweight CNN model to perform image modality feature extraction. Map time series feature modes and image feature modes to a unified hidden dimension.
[0009] Optionally, in one example of the above aspects, the bimodal feature alignment is divided into a distribution alignment stage, an attention semantic alignment stage, and a physiological association constraint stage. A generation stage strategy maps bimodal features to a unified feature space, including the following steps: The bimodal feature alignment is divided into three stages: distribution alignment, attention-semantic alignment, and physiological association constraint. In the distribution alignment stage of dual-modal feature alignment, the distribution offset between time series modal features and image modal features is eliminated by MMD maximum mean difference constraint, so that the distribution of time series modal features and image modal features is consistent. In the attention-semantic alignment stage of bimodal feature alignment, for bimodal features after distribution alignment, a bimodal attention matrix is constructed based on the Transformer attention mechanism, with time series nonlinear features as the core reference, and semantic alignment is performed. At the same time, for heart-related features, the corresponding attention weights are increased by 10% of the original attention weights; thus, the bimodal feature set after attention-semantic alignment is obtained.
[0010] In the physiological association constraint stage of bimodal feature alignment, a physiological association analysis term is introduced into the bimodal feature set after attention semantic alignment to analyze the correlation between the bimodal features after physiological association constraint and the heart state.
[0011] Optionally, in one example of the above aspects, a physiological correlation analysis term is introduced to analyze the correlation between the bimodal features after physiological correlation constraints and the cardiac state, including the following steps: A physiological association analysis term is introduced; features with a correlation coefficient greater than a threshold between bimodal features and cardiac state features are marked as features strongly associated with cardiac state.
[0012] Optionally, in one example of the above aspects, based on the bimodal features mapped to a unified feature space, a comprehensive strategy of adaptive modality fusion and multi-scale convolutional coding is established to perform deep co-coding of the bimodal features, outputting the final multi-scale co-coded features, including the following steps: Based on bimodal features mapped to a unified feature space, a comprehensive strategy for adaptive modality fusion and multi-scale convolutional coding is established, including: Adaptive modality fusion: By combining the feature parameters used to calculate the individual physiological characteristic assessment value P, an adaptive weight model for learning time series modality and image modality is established to dynamically adjust the contribution of the two modalities; Modality fusion of time series modality features and image modality features is performed based on the weights of time series modality and image modality; Multi-scale convolutional encoding: Through a multi-scale 1D convolutional module, it captures feature associations at different scales of modality fusion features. It contains three convolutional layers with different kernel sizes to capture short, medium and long-distance feature associations respectively, and outputs feature association vectors. Based on the output feature association vector, perform co-coding of dual-modal features to output the final multi-scale co-coded features.
[0013] Optionally, in one example of the above aspects, based on multi-scale co-coding features, a hierarchical multi-granularity annotation system is used to annotate the bimodal dataset. Using the annotated bimodal dataset as a basis, an SFT-supervised fine-tuning deep learning model is trained. The model with the highest accuracy is selected as the optimal model for the SFT stage. The multi-scale co-coding features are then analyzed, including the following steps: Based on the feature types of multi-scale collaborative coding features, the annotation hierarchy is divided into two levels: coarse-grained layer and fine-grained layer. In the coarse-grained layer, the time-series feature modalities and image modalities in the multi-scale co-coding features are classified and labeled as normal features or suspected abnormal features. In the fine-grained layer, the heart state strongly correlated features in the multi-scale co-encoding features are further subdivided into abnormal types, abnormal waveform features, or key event markings. The multi-scale collaborative coding features of historical data are labeled with coarse-grained and fine-grained layers; A CNN-RNN hybrid model is constructed as the basic architecture. The labeled multi-scale co-encoded features are divided into training and test sets. The CNN-RNN hybrid model is trained and the model is fine-tuned under supervision. During the test set validation process, the classification or regression ability of the model is comprehensively evaluated by accuracy, recall and F1 score. The performance of different fine-tuned models is evaluated, and the model with the highest weighted sum of accuracy, recall and F1 score is selected as the optimal model in the SFT stage. The optimal model in the SFT stage is used to analyze the multi-scale co-coding features to be detected, and the features are labeled in coarse-grained and fine-grained layers.
[0014] Optionally, in one example of the above aspects, based on the optimal model in the SFT stage, a data flywheel mechanism is introduced to perform data flywheel iteration and optimize the optimal model in the SFT stage, including the following steps: Based on the optimal model in the SFT stage, a data flywheel mechanism is introduced to continuously optimize the quality of training data, improve the performance of feature alignment and co-coding, and realize the self-evolution of the model, thus solving the limitation of existing technologies that rely on manual annotation and cannot continuously optimize data quality.
[0015] By using the optimal model in the SFT stage and introducing a data flywheel mechanism, the quality of training data is continuously optimized. The multi-scale co-coding features Ffinal of unlabeled multimodal ECG data are analyzed, and the analysis results and confidence scores Conf=softmax(Ffinal) are output. Set a confidence threshold, filter out unlabeled data with a confidence level greater than the threshold, and combine them with expert sampling review to include them in a high-quality training dataset; The newly added high-quality training data is fused with the original labeled data, the optimal model of the SFT stage is readjusted, and flywheel iteration is performed to obtain the model after flywheel iteration.
[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs a dual-modal dataset consisting of ECG time-series signals and 12-lead ECG images. It retains the advantage of ECG time-series signals in accurately capturing dynamic changes in cardiac electrical activity, while also preserving the spatial correlation features between the 12 leads through 12-lead standard image conversion. This overcomes the limitation of a single modality in comprehensively reflecting cardiac health, achieving full coverage of the two-dimensional features of cardiac electrical activity—both temporal dynamics and spatial distribution—providing a high-quality, multi-dimensional data foundation for subsequent feature alignment and co-coding. Simultaneously, targeted noise suppression is applied to both modal data, taking into account baseline drift, power line interference, and electromyographic noise in ECG time-series signals, as well as noise and interference residues in 12-lead ECG images. Simultaneously, modality adaptation operations such as standardization and time synchronization of the dual-modal data are completed, effectively reducing noise interference in subsequent feature extraction, alignment, and encoding, improving the purity and consistency of the dual-modal dataset, and avoiding feature deviations caused by improper modality adaptation. This provides a reliable guarantee for the smooth progress of subsequent technical steps.
[0017] This invention covers the entire process of multimodal data construction, feature alignment, co-coding, model training, and feature analysis, forming an end-to-end multimodal ECG processing technology chain. It is adaptable to ECG-Thinking benchmark-related application scenarios and possesses good compatibility and scalability. It comprehensively solves the core pain points of existing multimodal ECG feature alignment and co-coding, such as poor heterogeneous compatibility, insufficient coding discriminative power, weak scenario adaptability, and inaccurate feature analysis. This improves the utilization efficiency of multimodal ECG data, providing more accurate and reliable technical support for early diagnosis and dynamic monitoring of heart diseases, helping to reduce the rate of missed and misdiagnosed early heart diseases, and possesses significant clinical application value. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation
[0020] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] Please see Figure 1 The first aspect of this invention provides a method for feature alignment and collaborative coding of multimodal electrocardiogram data, comprising the following steps: An ECG time series was constructed, and the ECG time series signal was converted into a 12-lead standard image. The 12 channels correspond to the 12 leads, and the spatial correlation features between leads were preserved. In order to meet the requirements of ECG-Thinking multimodal processing, a dual-modal dataset of ECG time series signal combined with 12-lead ECG image was constructed, and targeted noise suppression was performed and modality adaptation was completed. We extract bimodal features from bimodal datasets and divide bimodal feature alignment into distribution alignment stage, attention semantic alignment stage, and physiological association constraint stage. We generate a stage strategy to map bimodal features to a unified feature space. Based on the dual-modal features mapped to a unified feature space, a comprehensive strategy of adaptive modality fusion and multi-scale convolutional coding is established to perform deep co-coding of the dual-modal features and output the final multi-scale co-coded features. Based on multi-scale co-coding features, a hierarchical multi-granularity labeling system is used to label the bimodal dataset. Based on the labeled bimodal dataset, a SFT-supervised fine-tuning deep learning model is trained, and the model with the highest accuracy is selected as the optimal model in the SFT stage. The multi-scale co-coding features are then analyzed. Based on the optimal model in the SFT stage, a data flywheel mechanism is introduced to perform data flywheel iteration and optimize the optimal model in the SFT stage.
[0022] Specifically, in this embodiment, a dual-modal dataset of ECG time-series signals and 12-lead ECG images is constructed. This not only retains the advantage of ECG time-series signals in accurately capturing dynamic changes in cardiac electrical activity, but also fully preserves the spatial correlation features between the 12 leads through 12-lead standard image conversion. This overcomes the limitation that a single modality cannot fully reflect the state of cardiac health, and achieves full coverage of the dual-dimensional features of cardiac electrical activity in terms of temporal dynamics and spatial distribution. This provides a high-quality, multi-dimensional data foundation for subsequent feature alignment and collaborative coding.
[0023] Simultaneously, targeted noise suppression was performed on the dual-modal data, taking into account baseline drift, power frequency interference, electromyographic motion noise in ECG time series signals, as well as noise and interference residues in 12-lead ECG images. At the same time, modality adaptation operations such as standardization and time synchronization of the dual-modal data were completed, effectively reducing the interference of noise on subsequent feature extraction, alignment and encoding, improving the purity and consistency of the dual-modal dataset, avoiding feature deviations caused by improper modality adaptation, and providing a reliable guarantee for the smooth progress of subsequent technical steps.
[0024] The construction of the bimodal dataset perfectly meets the requirements of ECG-Thinking multimodal processing and can be flexibly adapted to complex clinical monitoring scenarios such as frequent body movements, respiratory rhythm disorders, and comorbidities. It solves the problems of poor scenario adaptability and single feature dimension of existing single-modal datasets, and broadens the application scope of the method.
[0025] A phased feature alignment strategy, which combines distribution alignment, attention semantic alignment, and physiological association constraints, is adopted. This strategy breaks through the limitations of existing bimodal feature alignment, which only focuses on single-dimensional alignment and ignores semantic association and physiological meaning. It gradually achieves deep compatibility and accurate alignment of bimodal features, effectively solving the problems of heterogeneity, distribution shift, and semantic misalignment between ECG time series and 12-lead images.
[0026] The distribution alignment stage eliminates feature distribution bias caused by modal heterogeneity and noise by constraining the distribution consistency of bimodal features, ensuring that the bimodal features are initially in the same feature space. The attention-based semantic alignment stage relies on attention mechanisms, using core features of cardiac electrical activity as a reference to strengthen the semantic association of bimodal features, highlighting key features related to cardiac health and weakening redundant features. The physiological association constraint stage introduces physiological association constraints to ensure that the aligned bimodal features are strongly correlated with cardiac health status and have clear physiological meaning, avoiding the problem of aligned features being out of touch with clinical needs. Through this phased alignment strategy, bimodal features are accurately mapped to a unified feature space, achieving improved consistency and association of bimodal features. This provides high-quality aligned features for subsequent deep bimodal co-coding, solving the technical pain points of low accuracy and weak feature association in existing alignment methods, leading to poor coding performance.
[0027] Based on aligned features in a unified feature space, a comprehensive strategy of "adaptive modality fusion + multi-scale convolutional coding" is established. This strategy overcomes the limitations of existing modality fusion methods that use fixed weights and cannot adapt to individual and scene differences. Through adaptive modality weight learning, the contribution of time-series and image modalities can be dynamically adjusted according to individual physiological characteristics (complication complexity, body movement frequency, etc.), achieving personalization and adaptability of modality fusion and ensuring the reliability of encoded features in complex scenes. The multi-scale convolutional coding module captures short, medium, and long-distance correlations of dual-modality fusion features through different convolutional kernel sizes, effectively enhancing the robustness and discriminative power of the encoded features. It can accurately capture subtle nonlinear changes in early cardiac arrhythmias, solving the problems of existing coding methods being unable to effectively extract multi-scale features and having low sensitivity for early cardiac abnormality identification.
[0028] The dual-modal deep co-coding achieves the complementary advantages of time series nonlinear features and 12-lead image spatial features. It retains the ability of time series to capture dynamic changes in cardiac electrical activity, while also taking into account the ability of images to present spatial correlations between leads. The output multi-scale co-coding features have higher discriminative power and generalization ability, providing highly reliable feature support for subsequent cardiac disease diagnosis and abnormality identification.
[0029] A hierarchical multi-granularity annotation system is adopted to annotate the bimodal dataset. Compared with the existing single-granularity annotation, it can more meticulously and comprehensively characterize the correlation between bimodal features and cardiac health status, improve the effectiveness of the labeled data, provide a high-quality annotation foundation for the training of SFT supervised fine-tuning models, and help improve the learning accuracy of the models.
[0030] A supervised fine-tuned deep learning model using SFT was trained on a labeled bimodal dataset, and the model with the highest accuracy was selected as the optimal model. This ensured the model's ability to analyze multi-scale co-coded features, accurately uncovering the correlation between coded features and the type and severity of heart disease, enabling efficient interpretation and application of coded features. This addresses the problems of existing methods where coded features are disconnected from clinical diagnosis and cannot effectively analyze feature meaning. The optimal model in the SFT stage can accurately analyze multi-scale co-coded features and output the correlation logic between features and cardiac health status, providing interpretable evidence for clinical diagnosis, improving the clinical applicability of the method, and laying the foundation for further optimization and self-evolution of subsequent models.
[0031] In one embodiment of the present invention, an ECG time series is constructed, and the ECG time series signal is converted into a 12-lead standard image, with 12 channels corresponding to 12 leads, preserving the spatial correlation features between leads, including the following steps: An ECG time series is constructed, including core signals and auxiliary signals. The ECG time series is represented as Xts = {Xecg(t), Xres(t), Xmot(t), Xsp(t)}, where Xecg(t) is the original ECG time series at time t, and Xres(t), Xmot(t), and Xsp(t) are the auxiliary physiological signals at time t, where Xres(t) is the respiratory signal, Xmot(t) is the body movement signal, and Xsp(t) is the blood oxygenation signal. Set the size and resolution of the generated 12-lead standard image, construct a two-dimensional coordinate system for each lead's ECG signal, with the horizontal axis representing time and the vertical axis representing voltage, and discretize the preprocessed ECG signal into a series of points in chronological order, with each point corresponding to a time-voltage coordinate; The 12 leads are encoded according to the spatial coordinates between the leads. The 12-lead ECG includes standard limb leads (I, II, III, aVR, aVL, aVF) and chest leads (V1-V6). The 12 channels in the standard 12-lead image are set to correspond to the signals of the 12 leads. The channels corresponding to the standard limb leads are arranged in the upper half of the image, and the channels corresponding to the chest leads are arranged in the lower half of the image. In the coordinate system corresponding to each lead, the waveform of the ECG signal is plotted according to the discretized time-voltage coordinates. The lead code is added next to the waveform of each lead as an identifier to obtain the 12-lead standard image of the ECG time series signal conversion.
[0032] In one embodiment of the present invention, in response to the requirements of ECG-Thinking multimodal processing, a dual-modal dataset combining ECG time-series signals and 12-lead ECG images is constructed, targeted noise suppression is performed, and modality adaptation is completed, including the following steps: Median filtering is applied to the ECG time series signal, adaptive notch filtering is applied to the median-filtered ECG time series signal, and wavelet threshold denoising is applied to the filtered signal to suppress various types of noise in steps. The respiratory signal was filtered by moving average, the body movement signal was filtered by low-pass, and the blood oxygen signal was filtered by exponential smoothing to suppress auxiliary signal noise. Gaussian filtering was used to denoise the 12-lead standard image. The ECG time series signal was standardized by Z-score standardization, and the converted 12-lead standard image values were normalized. At the same time, the image size was adjusted to a uniform specification. A timestamp alignment algorithm is used to ensure that the time dimension of the standardized ECG time series signal is consistent with that of the normalized 12-lead standard image, thus completing modality adaptation and guaranteeing the temporal correlation of dual-modal features.
[0033] In one embodiment of the present invention, the ECG time series signal is subjected to median filtering, the median-filtered ECG time series signal is subjected to adaptive notch filtering, and wavelet threshold denoising is performed after filtering to suppress various types of noise in steps, including the following steps: Median filtering is applied to the ECG time series signal: Xecg1(t)=Median(Xecg(tw:t+w)), where w=round(fre×(0.05+0.03P)), where Xecg1(t) is the ECG time series after median filtering at time t, w is the median filtering window length, round(▪) is the rounding function, fre is the signal sampling frequency, P is the individual physiological characteristic assessment value, P=a1·Pcom+a2·Pmot+a3·Pres; Pcom is the comorbidity complexity, Pcom=number of comorbidities / base number of comorbidities, in this embodiment, the base number of comorbidities is preset to 30, Pmot is the body movement frequency, Pmot=number of activities in the detection period / number of basic activities in the detection period, in this embodiment, the number of basic activities in the detection period is preset to 40, Pres is the respiratory rhythm stability, Pres=1-(standard deviation of respiratory frequency / mean of respiratory frequency), a1, a2 and a3 are the corresponding weight coefficients; The ECG time series signal after median filtering is subjected to adaptive notch filtering: Xecg2(t) = Xecg1(t) - α(t)·Xecg1(t-τ), where Xecg2(t) is the ECG time series after adaptive notch filtering at time t, Xecg1(t-τ) is the ECG time series after median filtering at time t-τ, and τ is the delay time, τ = round(fre × 0.02), where 0.02 seconds corresponds to half a cycle of 50Hz power frequency interference (50Hz cycle is 0.02 seconds), and the power frequency noise is canceled out by the delay. α(t) is the adaptive adjustment coefficient. In this embodiment, the initial value of the adaptive adjustment coefficient α(t) is set to 0.4. The simplified adaptive algorithm based on minimum mean square error (LMS) updates α(t) iteratively to minimize the mean square error between the filtered signal and the reference noise, and then updates the adaptive adjustment coefficient α(t). After adaptive notch filtering, adaptive wavelet threshold denoising is used to suppress the superposition noise of electromyography and body movement as Xecg3(t) = Where Xecg3(t) is the ECG time series at time t with superimposed noise suppression of electromyography and body movement, J is the wavelet decomposition level of electromyography noise, and K is the wavelet decomposition level of body movement noise. The calculation formulas are J=5+round(P×2) and K=2+round(P×2), where P is the individual physiological characteristic assessment value. The stronger the body movement or electromyography noise (the larger P is), the more decomposition levels are required, and the more refined the separation of high-frequency noise can be achieved. Threshold λⱼ=σⱼ· The greater the intensity of the body motion signal, the higher the threshold, and the stronger the denoising effect; λj is the threshold of the wavelet coefficients of the j-th layer noise, σj is the standard deviation of the j-th layer noise, and N is the signal data length. The larger the data volume, the higher the threshold, to avoid excessive suppression of the effective signal. Let be the mean of the wavelet reconstruction coefficients of the wavelet decomposition of the j-th layer electromyographic noise and the corresponding wavelet decomposition of the k-th layer motion noise at the same instant. Let be the basis function of the j-th layer electromyographic noise wavelet decomposition, and the mean of the basis function of the k-th layer motion noise wavelet decomposition at time t.
[0034] In one embodiment of the present invention, extracting bimodal features from a bimodal dataset includes the following steps: Nonlinear and linear features are extracted to construct a time series feature set. For nonlinear features, the maximum Lyapunov exponent, correlation dimension, and sample entropy are extracted. For linear features, the mean / standard deviation of the RR interval and the amplitude / duration of the QRS wave are extracted. The formulas refer to the traditional linear feature calculation methods to extract the modal features of the time series. Spatial feature data of 12-lead standard images were extracted using a lightweight CNN model to perform image modality feature extraction; spatial correlation and waveform morphology features between leads were captured. Map time series feature modes and image feature modes to a unified hidden dimension.
[0035] In one embodiment of the present invention, bimodal feature alignment is divided into a distribution alignment stage, an attention semantic alignment stage, and a physiological association constraint stage. A generation stage strategy is used to map bimodal features to a unified feature space, including the following steps: Mapping time series features and image features to a unified hidden dimension The bimodal feature alignment is divided into three stages: distribution alignment, attention-semantic alignment, and physiological association constraint. In the distribution alignment stage of dual-modal feature alignment, the distribution offset between time series modal features and image modal features is eliminated by MMD maximum mean difference constraint, so that the distribution of time series modal features and image modal features is consistent; ensuring distribution consistency in a unified space and eliminating offset caused by noise and modal heterogeneity.
[0036] In the attention-semantic alignment stage of bimodal feature alignment, for bimodal features after distribution alignment, a bimodal attention matrix is constructed based on the Transformer attention mechanism, with time series nonlinear features as the core reference, and semantic alignment is performed. At the same time, for heart-related features, the corresponding attention weights are increased by 10% of the original attention weights; thus, the bimodal feature set after attention-semantic alignment is obtained.
[0037] In the physiological association constraint stage of bimodal feature alignment, a physiological association analysis term is introduced into the bimodal feature set after attention semantic alignment to analyze the correlation between the bimodal features after physiological association constraint and the heart state.
[0038] In one embodiment of the present invention, a physiological correlation analysis term is introduced to analyze the correlation between the bimodal features after physiological correlation constraints and the cardiac state, including the following steps: Introducing a physiological association analysis term, the formula is as follows: in, Let ρ(·,·) be the correlation coefficient between the bimodal features and the cardiac state features, and let ρ(·,·) be the Pearson correlation coefficient. Time series features with dual modal characteristics Image features related to the heart; The variance of each feature value in the h-th dimension after bimodal feature alignment. The values of each feature in the h-th dimension after bimodal feature alignment, h∈(1,2,…,H), where H is the total number of dimensions after bimodal feature alignment at each stage; Features whose correlation coefficient between bimodal features and cardiac state features is greater than a threshold are labeled as features strongly correlated with cardiac state.
[0039] In this embodiment, the heart-related features include: P wave: Represents atrial depolarization wave, and its morphology, amplitude, and duration can reflect the functional state of the atrium.
[0040] QRS complex: Represents ventricular depolarization waves. Its morphology, amplitude, and duration are of great significance in diagnosing heart diseases such as ventricular hypertrophy and myocardial infarction.
[0041] T wave: Represents the ventricular repolarization wave, and its morphology and direction can reflect the electrophysiological state of the ventricle.
[0042] ST segment: Represents the period from the end of ventricular depolarization to the beginning of repolarization. Its elevation or depression can reflect the blood supply status of the myocardium and is of great value in diagnosing myocardial ischemia and myocardial infarction.
[0043] U wave: Influenced by the subsequent potential, its appearance and morphological changes may be related to cardiac electrophysiological abnormalities.
[0044] PR interval: The time from the start of atrial depolarization to the start of ventricular depolarization, which reflects the functional state of the atrioventricular conduction system.
[0045] QRS interval: Represents the time required for ventricular depolarization, and has auxiliary value in diagnosing heart diseases such as ventricular hypertrophy and myocardial infarction.
[0046] QT interval: represents the time required for the entire process of ventricular depolarization and repolarization. Its prolongation may be related to cardiac electrophysiological abnormalities.
[0047] In one embodiment of the present invention, based on bimodal features mapped to a unified feature space, a comprehensive strategy of adaptive modality fusion and multi-scale convolutional coding is established to perform deep co-coding of bimodal features, preserving the nonlinear features of the time series and the spatial features of the image, and outputting the final multi-scale co-coded features, including the following steps: Based on bimodal features mapped to a unified feature space, a comprehensive strategy for adaptive modality fusion and multi-scale convolutional coding is established, including: Adaptive modality fusion: Combining feature parameters used to calculate the individual physiological characteristic assessment value P, an adaptive weight model for learning time-series modality and image modality is established to dynamically adjust the contribution of the two modalities. The formula is as follows: Wts=σ(sts), Wimg=σ(simg); Where σ(·) is the sigmoid function, Wts is the weight of the time series modality, reflecting the importance of time series data (such as electrocardiogram time series, which reflects changes in cardiac electrical activity over time) in the overall bimodal analysis. It is calculated using σ(sts) and ranges from 0 to 1. Wimg is the weight of the image modality, representing the importance of image data (such as echocardiogram images, which show information about cardiac structure) in the bimodal analysis. It is also calculated using σ(simg) and ranges from 0 to 1.
[0048] sts is the cosine similarity between the time series modal features output by the intermediate layer of the adaptive weighting model and the individual physiological features; simg is the cosine similarity between the image modal features output by the intermediate layer of the adaptive weighting model and the individual physiological features, used to calculate the weights of the image modal; Wts + Wimg = 1; the individual physiological features, i.e. the feature parameters used to calculate the individual physiological feature evaluation value P, include: comorbidity complexity, body movement frequency, and respiratory rhythm stability. Modality fusion of time series modality features and image modality features is performed based on the weights of time series modality and image modality; Multi-scale convolutional encoding: A multi-scale 1D convolutional module captures feature associations at different scales of modality fusion features. This includes three convolutional layers with different kernel sizes (3, 5, and 7) to capture short, medium, and long-distance feature associations respectively, outputting feature association vectors. Furthermore, in this embodiment, a batch normalization (BN) layer is introduced to accelerate the convergence of the subsequent three-stage training, improving the model's generalization ability and laying the foundation for state-of-the-art (SOTA) performance.
[0049] Based on the output feature association vector, dual-modal feature co-encoding is performed to output the final multi-scale co-encoded features. These features are used for subsequent three-stage training and expert-level inference, while also adapting to individual physiological differences.
[0050] In one embodiment of the present invention, based on multi-scale co-coding features, a hierarchical multi-granularity annotation system is used to annotate the bimodal dataset. Using the annotated bimodal dataset as a basis, a SFT-supervised fine-tuning deep learning model is trained. The model with the highest accuracy is selected as the optimal model for the SFT stage. The multi-scale co-coding features are then analyzed, including the following steps: analyzing time-series feature modalities and image modalities, and identifying strong correlation features with cardiac status. Based on the feature types of multi-scale collaborative coding features, the annotation hierarchy is divided into two levels: coarse-grained layer and fine-grained layer. In the coarse-grained layer, the time-series feature modalities and image modalities in the multi-scale co-coding features are classified and labeled as normal features or suspected abnormal features. In the fine-grained layer, the heart state strongly correlated features in the multi-scale co-encoding features are further subdivided into abnormal types and labeled as abnormal waveform features or key event markers. The multi-scale collaborative coding features of historical data are labeled with coarse-grained and fine-grained layers; A CNN-RNN hybrid model is constructed as the basic architecture. The labeled multi-scale co-encoded features are divided into training and test sets. The CNN-RNN hybrid model is trained and the model is fine-tuned under supervision. During the test set validation process, the classification or regression ability of the model is comprehensively evaluated by accuracy, recall and F1 score. The performance of different fine-tuned models is evaluated, and the model with the highest weighted sum of accuracy, recall and F1 score is selected as the optimal model in the SFT stage. By using the optimal model in the SFT stage, the multi-scale co-coding features to be detected are analyzed, and the features are labeled at both coarse-grained and fine-grained levels.
[0051] In one embodiment of the present invention, based on the optimal model of the SFT stage, a data flywheel mechanism is introduced to perform data flywheel iteration and optimize the optimal model of the SFT stage, including the following steps: Phase 2: Data Flywheel Iteration (Self-evolution, optimizing data quality) Based on the optimal model in the SFT stage, a data flywheel mechanism is introduced to continuously optimize the quality of training data, improve the performance of feature alignment and co-coding, and realize the self-evolution of the model, thus solving the limitation of existing technologies that rely on manual annotation and cannot continuously optimize data quality.
[0052] By using the optimal model in the SFT stage and introducing a data flywheel mechanism, the quality of training data is continuously optimized. The multi-scale co-coding features Ffinal of unlabeled multimodal ECG data are analyzed, and the analysis results and confidence scores Conf=softmax(Ffinal) are output. Set a confidence threshold, filter out unlabeled data with a confidence level greater than the threshold, and combine it with expert sampling review (the review ratio is set to 10%) to include it in the high-quality training dataset. The newly added high-quality training data is fused with the original labeled data, the optimal model of the SFT stage is readjusted, and flywheel iteration is performed to obtain the model after flywheel iteration.
[0053] Perform closed-loop iteration, repeating steps 1-3, completing one flywheel iteration every 30 days, with each flywheel iteration consisting of 20 rounds, continuously optimizing the quality of training data and the performance of the encoding model, and achieving model self-evolution.
[0054] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.
Claims
1. A method for feature alignment and collaborative coding of multimodal electrocardiogram data, characterized in that, Includes the following steps: An ECG time series was constructed, and the ECG time series signal was converted into a 12-lead standard image. The 12 channels correspond to the 12 leads, and the spatial correlation features between leads were preserved. In order to meet the requirements of ECG-Thinking multimodal processing, a dual-modal dataset of ECG time series signal combined with 12-lead ECG image was constructed, and targeted noise suppression was performed and modality adaptation was completed. We extract bimodal features from bimodal datasets and divide bimodal feature alignment into distribution alignment stage, attention semantic alignment stage, and physiological association constraint stage. We generate a stage strategy to map bimodal features to a unified feature space. Based on the dual-modal features mapped to a unified feature space, a comprehensive strategy of adaptive modality fusion and multi-scale convolutional coding is established to perform deep co-coding of the dual-modal features and output the final multi-scale co-coded features. Based on multi-scale co-coding features, a hierarchical multi-granularity labeling system is used to label the bimodal dataset. Based on the labeled bimodal dataset, a SFT-supervised fine-tuning deep learning model is trained, and the model with the highest accuracy is selected as the optimal model in the SFT stage. The multi-scale co-coding features are then analyzed. Based on the optimal model in the SFT stage, a data flywheel mechanism is introduced to perform data flywheel iteration and optimize the optimal model in the SFT stage.
2. The feature alignment and collaborative coding method for multimodal electrocardiogram data according to claim 1, characterized in that, Constructing an ECG time series involves converting the ECG time series signal into a 12-lead standard image, with 12 channels corresponding to 12 leads, preserving the spatial correlation features between leads, including the following steps: Construct an ECG time series, including core and auxiliary signals; Set the size and resolution of the generated 12-lead standard image, construct a two-dimensional coordinate system for each lead's ECG signal, with the horizontal axis representing time and the vertical axis representing voltage, and discretize the preprocessed ECG signal into a series of points in chronological order, with each point corresponding to a time-voltage coordinate; The 12-lead ECG is encoded based on the spatial coordinates between the leads. The 12-lead ECG includes standard limb leads and chest leads. The 12 channels in the standard 12-lead image are set to correspond to the signals of the 12 leads. The channels corresponding to the standard limb leads are arranged in the upper half of the image, and the channels corresponding to the chest leads are arranged in the lower half of the image. In the coordinate system corresponding to each lead, the waveform of the ECG signal is plotted according to the discretized time-voltage coordinates. The lead code is added next to the waveform of each lead as an identifier to obtain the 12-lead standard image of the ECG time series signal conversion.
3. The feature alignment and collaborative coding method for multimodal electrocardiogram data according to claim 1, characterized in that, To address the requirements of ECG-Thinking multimodal processing, a dual-modal dataset combining ECG time-series signals and 12-lead ECG images was constructed. Targeted noise suppression and modality adaptation were then performed, including the following steps: The ECG time series signal is subjected to median filtering, the median-filtered ECG time series signal is subjected to adaptive notch filtering, and wavelet threshold denoising is performed after filtering to suppress various types of noise in steps. The respiratory signal was filtered by moving average, the body movement signal was filtered by low-pass, and the blood oxygen signal was filtered by exponential smoothing to suppress auxiliary signal noise. Gaussian filtering was used to denoise the 12-lead standard image. The ECG time series signal was standardized by Z-score standardization, and the converted 12-lead standard image values were normalized. At the same time, the image size was adjusted to a uniform specification. By using a timestamp alignment algorithm, the time dimension of the standardized ECG time series signal is made consistent with that of the normalized 12-lead standard image, thus completing modality adaptation.
4. The feature alignment and collaborative coding method for multimodal electrocardiogram data according to claim 3, characterized in that, The ECG time series signal is subjected to median filtering, followed by adaptive notch filtering, and then wavelet threshold denoising to suppress various types of noise in stages, including the following steps: Median filtering is applied to the ECG time series signal: Xecg1(t)=Median(Xecg(tw:t+w)), where w=round(fre×(0.05+0.03P)), where Xecg1(t) is the ECG time series after median filtering at time t, w is the median filtering window length, round(▪) is the rounding function, fre is the signal sampling frequency, P is the individual physiological characteristic assessment value, P=a1·Pcom+a2·Pmot+a3·Pres; Pcom is the comorbidity complexity, Pcom=number of comorbidities / base number of comorbidities, Pmot is the body movement frequency, Pmot=number of activities in the detection period / number of basic activities in the detection period, Pres is the respiratory rhythm stability, Pres=1-(standard deviation of respiratory frequency / mean of respiratory frequency), a1, a2 and a3 are the corresponding weight coefficients; The ECG time series signal after median filtering is subjected to adaptive notch filtering: Xecg2(t)=Xecg1(t)-α(t)·Xecg1(t-τ), where Xecg2(t) is the ECG time series after adaptive notch filtering at time t, Xecg1(t-τ) is the ECG time series after median filtering at time t-τ, τ is the delay time, τ=round(fre×0.02), and α(t) is the adaptive adjustment coefficient; After adaptive notch filtering, adaptive wavelet threshold denoising is used to suppress the superposition noise of electromyography and body movement as Xecg3(t) = Where Xecg3(t) is the ECG time series with superimposed noise suppression of electromyography and body movement at time t, J is the wavelet decomposition level of electromyography noise, K is the wavelet decomposition level of body movement noise, and the calculation formulas are J=5+round(P×2), K=2+round(P×2), P is the individual physiological characteristic assessment value, and the threshold λⱼ=σⱼ· The greater the intensity of the body motion signal, the larger the threshold, and the stronger the denoising effect; λj is the threshold of the wavelet coefficients of the j-th layer of noise, σj is the standard deviation of the j-th layer of noise, and N is the signal data length. Let be the mean of the wavelet reconstruction coefficients of the wavelet decomposition of the j-th layer electromyographic noise and the corresponding wavelet decomposition of the k-th layer motion noise at the same instant. Let be the basis function of the j-th layer electromyographic noise wavelet decomposition, and the mean of the basis function of the k-th layer motion noise wavelet decomposition at time t.
5. A feature alignment and cooperative coding method for multimodal electrocardiogram data according to claim 1, characterized in that, Extracting bimodal features from a bimodal dataset includes the following steps: Extract nonlinear and linear features to construct a time series feature set; for nonlinear features, extract the maximum Lyapunov exponent, correlation dimension, and sample entropy; for linear features, extract the mean / standard deviation of RR intervals and QRS amplitude / duration to extract time series modal features. Spatial feature data of 12-lead standard images were extracted using a lightweight CNN model to perform image modality feature extraction. Map time series feature modes and image feature modes to a unified hidden dimension.
6. The feature alignment and collaborative coding method for multimodal electrocardiogram data according to claim 1, characterized in that, The bimodal feature alignment is divided into three stages: distribution alignment, attention-semantic alignment, and physiological association constraint. A generation-stage strategy maps bimodal features to a unified feature space, including the following steps: The bimodal feature alignment is divided into three stages: distribution alignment, attention-semantic alignment, and physiological association constraint. In the distribution alignment stage of dual-modal feature alignment, the distribution offset between time series modal features and image modal features is eliminated by MMD maximum mean difference constraint, so that the distribution of time series modal features and image modal features is consistent. In the attention-semantic alignment stage of bimodal feature alignment, for bimodal features after distribution alignment, a bimodal attention matrix is constructed based on the Transformer attention mechanism, with time series nonlinear features as the core reference, and semantic alignment is performed. At the same time, for heart-related features, the corresponding attention weights are increased by 10% of the original attention weights; thus, the bimodal feature set after attention-semantic alignment is obtained. In the physiological association constraint stage of bimodal feature alignment, a physiological association analysis term is introduced into the bimodal feature set after attention semantic alignment to analyze the correlation between the bimodal features after physiological association constraint and the heart state.
7. A feature alignment and collaborative coding method for multimodal electrocardiogram data according to claim 6, characterized in that, Introducing a physiological correlation analysis term, we analyze the correlation between the bimodal features after physiological correlation constraints and cardiac status, including the following steps: Introducing a physiological association analysis term, the formula is as follows: in, Let ρ(·,·) be the correlation coefficient between the bimodal features and the cardiac state features, and let ρ(·,·) be the Pearson correlation coefficient. Time series features with dual modal characteristics Image features related to the heart; The variance of each feature value in the h-th dimension after bimodal feature alignment. The values of each feature in the h-th dimension after bimodal feature alignment, h∈(1,2,…,H), where H is the total number of dimensions after bimodal feature alignment at each stage; Features whose correlation coefficient between bimodal features and cardiac state features is greater than a threshold are labeled as features strongly correlated with cardiac state.
8. A feature alignment and collaborative coding method for multimodal electrocardiogram data according to claim 1, characterized in that, Based on bimodal features mapped to a unified feature space, a comprehensive strategy of adaptive modality fusion and multi-scale convolutional coding is established to perform deep co-coding of bimodal features and output the final multi-scale co-coded features, including the following steps: Based on bimodal features mapped to a unified feature space, a comprehensive strategy for adaptive modality fusion and multi-scale convolutional coding is established, including: Adaptive modality fusion: Combining feature parameters used to calculate the individual physiological characteristic assessment value P, an adaptive weight model for learning time-series modality and image modality is established to dynamically adjust the contribution of the two modalities. The formula is as follows: Wts=σ(sts), Wimg=σ(simg); Where σ(·) is the sigmoid function, Wts is the weight of the time series mode, and Wimg is the weight of the image mode; sts is the cosine similarity between the time-series modal features output by the intermediate layer of the adaptive weighting model and the individual physiological features; simg is the cosine similarity between the image modal features output by the intermediate layer of the adaptive weighting model and the individual physiological features, used to calculate the weights of the image modal; Wts + Wimg = 1; the individual physiological features are the feature parameters used to calculate the individual physiological feature evaluation value P, including: comorbidity complexity, body movement frequency, and respiratory rhythm stability. Modality fusion of time series modality features and image modality features is performed based on the weights of time series modality and image modality; Multi-scale convolutional encoding: Through a multi-scale 1D convolutional module, it captures feature associations at different scales of modality fusion features. It contains three convolutional layers with different kernel sizes to capture short, medium and long-distance feature associations respectively, and outputs feature association vectors. Based on the output feature association vector, perform co-coding of dual-modal features to output the final multi-scale co-coded features.
9. A feature alignment and collaborative coding method for multimodal electrocardiogram data according to claim 1, characterized in that, Based on multi-scale co-coding features, a hierarchical multi-granularity annotation system is used to annotate the bimodal dataset. Using the annotated bimodal dataset as a foundation, a Deep Learning Model with Supervised Fine-tuning (SFT) is trained. The model with the highest accuracy is selected as the optimal model for the SFT stage. The multi-scale co-coding features are then analyzed, including the following steps: Based on the feature types of multi-scale collaborative coding features, the annotation hierarchy is divided into two levels: coarse-grained layer and fine-grained layer. In the coarse-grained layer, the time-series feature modalities and image modalities in the multi-scale co-coding features are classified and labeled as normal features or suspected abnormal features. In the fine-grained layer, the strongly correlated cardiac state features in the multi-scale co-coding features are further subdivided into abnormal waveform features or key event markers. The multi-scale collaborative coding features of historical data are labeled with coarse-grained and fine-grained layers; A CNN-RNN hybrid model is constructed as the basic architecture. The labeled multi-scale co-encoded features are divided into training and test sets. The CNN-RNN hybrid model is trained and the model is fine-tuned under supervision. During the test set validation process, the classification or regression ability of the model is comprehensively evaluated by accuracy, recall and F1 score. The performance of different fine-tuned models is evaluated, and the model with the highest weighted sum of accuracy, recall and F1 score is selected as the optimal model in the SFT stage. By using the optimal model in the SFT stage, the multi-scale co-coding features to be detected are analyzed, and the features are labeled at both coarse-grained and fine-grained levels.
10. A feature alignment and cooperative coding method for multimodal electrocardiogram data according to claim 1, characterized in that, Based on the optimal model in the SFT stage, a data flywheel mechanism is introduced to perform data flywheel iteration and optimize the optimal model in the SFT stage, including the following steps: Based on the optimal model in the SFT stage, a data flywheel mechanism is introduced to continuously optimize the quality of training data, improve the performance of feature alignment and co-coding, realize the self-evolution of the model, and solve the limitation of existing technologies that rely on manual annotation and cannot continuously optimize data quality. By using the optimal model in the SFT stage and introducing a data flywheel mechanism, the quality of training data is continuously optimized. The multi-scale co-coding features Ffinal of unlabeled multimodal ECG data are analyzed, and the analysis results and confidence scores Conf=softmax(Ffinal) are output. Set a confidence threshold, filter out unlabeled data with a confidence level greater than the threshold, and combine them with expert sampling review to include them in a high-quality training dataset; The newly added high-quality training data is fused with the original labeled data, the optimal model of the SFT stage is readjusted, and flywheel iteration is performed to obtain the model after flywheel iteration.