Electrocardiogram paper tape data base model construction system based on multi-mode comparative learning

Abstract

The invention discloses an electrocardiogram paper tape data base model construction system based on multi-modal comparative learning. The system comprises an image sample pair signal generation module, a multi-modal self-supervision pre-training model construction module and an intelligent image cutting module. The signal conversion module is used for converting an original ECG signal into a standardized signal and a matched image, and outputting an image sample pair signal of a sample pair data set; meanwhile, an image sample pair signal is received, an image encoder is trained through multi-modal contrast learning, and a trained electrocardiogram paper tape data base model is output; in addition, the real scene ECG scans are converted into standardized input images. According to the system, standardized ECG image-digital signal sample pairs are generated, a pre-trained and frozen ECGFounder signal model is constructed to guide a ViT visual model to learn a cross-modal contrast learning model of ECG image semantics, an intelligent cutting module is developed to adapt to a real and complex ECG scanning copy, and the model does not need to be trained again.

Description

A System for Constructing ECG Tape Data Base Model Based on Multimodal Comparative Learning Technical Field

[0001] This invention relates to the field of machine learning technology, specifically to a system for constructing an electrocardiogram (ECG) tape data base model based on multimodal contrastive learning. Background Technology

[0002] Cardiovascular diseases (CVDs) have become the leading health threat globally, placing a heavy burden on public health systems. Electrocardiography (ECG), as a non-invasive, economical, and widely used diagnostic tool, plays a central role in the screening, diagnosis, and monitoring of CVDs. A standard 12-lead ECG captures cardiac electrical activity from a multi-dimensional perspective, providing crucial information for clinical diagnosis. However, accurate ECG interpretation highly depends on experienced physicians, and in many regions, especially at the grassroots level, a shortage of qualified personnel leads to low diagnostic efficiency and even the risk of misdiagnosis. In recent years, artificial intelligence technologies, represented by deep learning, have made significant progress in automated ECG analysis, with performance comparable to human experts in certain specific tasks. However, these supervised learning models generally face a "dual bottleneck" of data and annotation: on the one hand, obtaining large-scale, diverse medical data is limited by privacy regulations; on the other hand, high-quality expert annotation is costly and time-consuming, severely restricting the further development and clinical application of AI technology.

[0003] Traditional automated ECG analysis heavily relies on high-quality digital time-series signals. Whether it's early rule-based and statistical feature-based methods or modern deep learning models based on one-dimensional convolutional neural networks (1D-CNN) or recurrent neural networks (RNN), the cornerstone of their success is voltage sequences accurate to the microvolt level. Early automated methods primarily depended on sophisticated signal processing techniques and manual feature engineering. For example, the classic algorithm proposed by Pan and Tompkins et al. achieved robust detection of QRS complexes through a series of steps including differencing, squaring, integrating, and adaptive thresholding, laying the foundation for subsequent heart rate variability (HRV) analysis and preliminary arrhythmia assessment. Building on this, researchers further utilized techniques such as wavelet transform to extract morphological features, temporal intervals, and frequency energy distributions of the P-QRS-T waves from ECG signals. These manually designed features were then fed into traditional machine learning classifiers such as support vector machines (SVM) or random forests for diagnostic purposes.

[0004] With the rise of deep learning, end-to-end learning paradigms have gradually replaced cumbersome feature engineering. Kiranyaz et al. pioneered the systematic application of one-dimensional convolutional neural networks (1D-CNN) to ECG classification, demonstrating that CNNs can automatically learn more discriminative morphological patterns than handcrafted features. A landmark work in this direction came from Rajpurkar et al., whose deep 1D-CNN model achieved cardiologist-level performance on arrhythmia classification tasks, fully demonstrating the powerful ability of CNNs to capture local morphological features of ECG waveforms. Meanwhile, to better model the inherent temporal dependencies of ECG signals, Faust et al. explored the use of Long Short-Term Memory (LSTM) networks to analyze heartbeat sequences, identifying different types of arrhythmias by capturing variations in the RR interval. To combine the advantages of these two models, Schwab et al. proposed a hybrid architecture combining CNNs and RNNs. First, deep morphological features of each heartbeat are extracted using CNN layers, and then these feature sequences are input into RNN layers for temporal modeling, thus simultaneously considering information from both morphological and rhythmic dimensions. Recently, inspired by the field of natural language processing, Natarajan et al. introduced the Transformer architecture into ECG analysis, utilizing its self-attention mechanism to capture long-distance dependencies between any two time points in the signal, providing a new perspective for understanding complex, cross-cardiac pathological patterns. Although these methods have continuously evolved in architecture and become increasingly powerful, they all invariably require high-fidelity digital signals as input, making them inadequate when faced with massive ECG image archives.

[0005] To bridge this gap, a seemingly intuitive approach is "curve reconstruction": converting images into signals, tracing waveform curves from ECG drawings, converting them back into one-dimensional digital signals, and then applying mature signal analysis models. This research direction, known as "ECG digitization," has attracted numerous scholars. For example, Xiongjun Zhao et al. proposed the ECG-I2S method, which, through a series of processes including image preprocessing, signal extraction, heartbeat cycle division, and dimensionality reduction, can accurately extract one-dimensional numerical signals from ECG images. However, we believe that the "ECG digitization" approach is fundamentally flawed. ECG images are lossy representations of raw electrical signals after multiple stages of printing, storage, and scanning. Issues such as uneven lighting, paper wrinkles, background grid interference, ink breaks, and baseline drift during scanning can cause irreversible damage to the waveform. Any attempt to perfectly reconstruct a precise one-dimensional signal from such noisy two-dimensional data will inevitably introduce interpolation errors and artifacts, leading to signal distortion. No matter how powerful the subsequent analysis model is, the reliability of its results will be greatly reduced based on these "contaminated" inputs, creating a "garbage in, garbage out" dilemma. This method is not only complex and fragile, but also discards the visual contextual information that the image itself may contain.

[0006] Meanwhile, the ECG field is witnessing a surge in "foundation models." Inspired by pre-trained models in natural language processing (such as BERT), researchers have begun training foundation models, such as ECGFounder and KED, on massive amounts of unlabeled ECG digital signals. These models, through self-supervised learning, have acquired a deep understanding of the intrinsic structure, rhythm, and morphology of ECG signals, generating powerful, transferable feature representations. This advancement offers an unprecedented opportunity. Instead of struggling with the treacherous "image-to-signal" conversion path, it's possible to directly train an image model using a large model. Therefore, the question arises: can a visual model be directly taught to "read" ECG images, achieving an understanding comparable to top-tier signal foundation models?

[0007] The current field of automated electrocardiogram (ECG) analysis suffers from a fundamental paradigmatic limitation: both early signal processing methods and modern deep learning models are highly focused on processing one-dimensional digital time-series signals. Even seemingly image-oriented solutions, such as "ECG digitization," are essentially extensions of signal-centric thinking—their goal is to painstakingly convert two-dimensional images back into one-dimensional signals so that mature signal analysis techniques can still be applied. This paradigm leads to a significant contradiction: on the one hand, AI models are becoming increasingly powerful at the signal level; on the other hand, in the real world, most doctors and patients are exposed to printed two-dimensional ECG images, and hospitals worldwide store hundreds of millions of valuable historical medical records in paper or scanned form. This massive amount of image data, due to its incompatibility with mainstream AI models, has become an ineffective "information silo," severely hindering the widespread adoption and application of AI technology in real-world clinical settings. Summary of the Invention

[0008] The purpose of this invention is to provide a system for constructing an electrocardiogram (ECG) paper tape data base model based on multimodal contrastive learning, in order to solve the technical problems existing in the prior art.

[0009] The system comprises three parts: an image sample pair signal generation module, a multimodal self-supervised pre-training model construction module, and an intelligent image cropping module. The image sample pair signal generation module converts the raw ECG signal into a standardized signal and matching image, outputting the image sample pair signal of the sample pair dataset, which is then input into the multimodal self-supervised pre-training model construction module. The multimodal self-supervised pre-training model construction module receives the image sample pair signal, trains the image encoder through multimodal contrastive learning, and outputs a trained ECG paper tape data base model. The intelligent image cropping module converts real-world ECG scans into standardized input images, providing adapted input for the image model without requiring model retraining.

[0010] Preferably, the image sample pair signal generation module performs a standardized preprocessing procedure on the original one-dimensional ECG signal to remove baseline drift and interference noise.

[0011] Preferably, the removal of baseline drift and interference noise specifically includes: (1) Filtering: Applying a 4th-order Butterworth bandpass filter to remove baseline drift and high-frequency noise while retaining the key components of the ECG waveform; (2) Length normalization: All signals are truncated or padded to a fixed length; (3) Amplitude normalization: Z-score normalization is performed on each of the 12 leads; (4) Image rendering: First, a plotting module is customized based on the open-source ecg_plot library, and image rendering is performed in accordance with the clinical ECG display standard. Then, the 12 leads are arranged in a grid layout of 4 columns and 3 rows, with a background containing a standard red millimeter grid. Finally, the aspect ratio and DPI of the image are accurately calculated to generate a clean, high-resolution PNG image without background borders or titles; (5) A dataset of image and signal matching pairs of ECG images and index files that are matched one-to-one with the normalized signals is formed, i.e., high-quality image sample pairs of signals.

[0012] Preferably, the processing steps of the multimodal self-supervised pre-training model construction module include: (1) model architecture design, forming an initial model architecture, including a frozen signal encoder, a trainable image encoder and an image projection head; (2) constructing an ECG paper tape data base model based on multimodal contrastive learning.

[0013] Preferably, the intelligent image cropping module is based on a series of classic computer vision techniques, including adaptive thresholding, morphological image processing and contour analysis, and combines these techniques into an intelligent cropping process based on a hybrid strategy.

[0014] Preferably, the intelligent cropping process based on the hybrid strategy specifically includes: (1) macro-content positioning: finding the content outline with the largest area in the image through adaptive threshold and morphological processing; (2) refined hybrid cropping: setting a grid height cropping ratio on the vertical axis, retaining only 78% of the height of the initial cropping area starting from the top; (3) size standardization: scaling the finally cropped image to a uniform size.

[0015] Preferably, the image projection head is used to calculate the contrast loss in the same space. A simple linear layer is added after the image encoder to linearly transform the original output features to a space consistent with the signal embedding dimension, resulting in the final image embedding used for contrast.

[0016] Preferably, the construction of the ECG paper tape data base model based on multimodal contrastive learning specifically includes: (1) determining the contrastive learning scheme and objectives; (2) configuring experimental and training parameters; (3) determining training convergence and terminating pre-training; and (4) model decoupling and instantiation.

[0017] Preferably, the step of converting real-scene ECG scans into standardized input images specifically includes: (obtaining signal embeddings and image embeddings through their respective encoders and projectors, setting the image embedding corresponding to each signal embedding as a positive sample, setting other image embeddings in the batch as negative samples, designing a symmetrical loss function, simultaneously optimizing the alignment in both the "signal to image" and "image to signal" directions, calculating the loss from image to signal and the loss from signal to image, and then taking the average of the two to obtain the total contrast loss.)

[0018] Preferably, image-to-signal loss : Signal-to-image loss : ;in, It is cosine similarity. It is a learnable temperature parameter that controls the sharpness of the softmax function and is crucial to the stability of the learning process. Total contrast loss. The total comparative loss is the average of the losses in these two directions, which is then summed and averaged over the entire batch. .

[0019] The method of the present invention has the following advantages: (1) It proposes an efficient data generation strategy to convert digital signals into standardized ECG images and generate sample pairs in batches, laying the foundation for cross-modal learning.

[0020] (2) Design a cross-modal contrastive learning model that uses a pre-trained, frozen signal foundation model ECGFounder to guide a visual model ViT in learning the semantics of ECG images. (3) Develop an intelligent cropping module that enables our trained model to handle complex real-world ECG scans without retraining. Attached Figure Description

[0021] Figure 1 is a schematic diagram of the overall model framework; Figure 2 is a schematic diagram of the signal encoder network structure; Figure 3 is a schematic diagram of the image encoder network structure; Figure 4 is a schematic diagram of the intelligent cropping module workflow; Figure 5 is a schematic diagram of the contrastive learning architecture; Figure 6 is a flowchart of downstream tasks; Figure 7 is a diagram of ROC AUC results on PTB-XL; Figure 8 is a diagram of ROC AUC results on the National Heart Foundation 2023 ECG dataset. Detailed Implementation

[0022] The technical solution of the present invention will be clearly and completely described below with reference to specific embodiments. However, those skilled in the art should understand that the embodiments described below are only for illustrating the present invention and should not be regarded as limiting the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] This invention provides a system for constructing an electrocardiogram (ECG) paper tape data base model based on multimodal contrastive learning.

[0024] This system involves a novel, image-oriented analysis paradigm aimed at training a visual model capable of directly "reading" and understanding ECG images like a human expert, achieving semantic understanding comparable to top-tier signal-based models. It was observed that any ECG record, its one-dimensional time-series signal and two-dimensional printed image, are two different representations of the same underlying physiological information. They are semantically equivalent. Therefore, the original ECG signal can be considered a natural, cost-free, and highly information-dense supervisory signal for its corresponding image. Based on this idea, this invention proposes a cross-modal contrastive learning framework. It can operate without relying on any disease labels, instead setting a pretext task: training an image model to "read" the same semantic information from an ECG image as the original signal. Specifically, the model needs to learn a mapping function that projects the ECG image onto a feature vector. This vector should be closely aligned with the feature vector of its corresponding signal in the embedding space, while being mutually exclusive with the feature vectors of other unrelated samples within the batch. By solving this alignment problem, the image model can autonomously learn key visual patterns in ECG images, such as the morphology of the P wave, QRS complex, and T wave, ST segment deviation, and the regularity of the heart rhythm. These are the core bases for human experts to make diagnoses.

[0025] The system comprises three parts: an image sample pair signal generation module, a multimodal self-supervised pre-trained model construction module, and an intelligent image cropping module. As shown in Figure 1, these three modules form an input-processing-output closed-loop architecture. The specific processing steps of each module are as follows: The image sample pair signal generation module converts the original ECG signal into a standardized signal and a matching image, outputting the image sample pair signal of the sample pair dataset, which is then input into the multimodal self-supervised pre-trained model construction module.

[0026] In order to obtain high-quality image sample pair signals, the image sample pair signal generation module performs a standardized preprocessing procedure on the original one-dimensional ECG signal to remove baseline drift and interference noise. Specifically, this includes: (1) Filtering: Applying a fourth-order Butterworth bandpass filter with a passband range of 0.5Hz to 45.0Hz. This effectively removes baseline drift and high-frequency noise while retaining the key components of the ECG waveform.

[0027] (2) Length normalization: All signals are truncated or padded to a fixed length (1250 sampling points, 500 Hz sampling rate) of 2.5 s. If the original record is less than 2.5 seconds, zero padding is performed at the end.

[0028] (3) Amplitude normalization: Z-score normalization was performed on each of the 12 leads (channels). The mean and standard deviation of each channel were pre-calculated on the entire training set to ensure the consistency of scaling.

[0029] (4) Image rendering: First, a plotting module was customized based on the open-source ecg_plot library, and the image was rendered in accordance with the clinical ECG display standards: the paper feed speed was set to 25 mm / s, and the voltage gain (amplitude) was set to 10 mm / mV; then the 12 leads were arranged in a grid layout of 4 columns and 3 rows ('I','II','III','aVR','aVL','aVF','V1','V2','V3','V4','V5','V6'), with the background containing a standard red millimeter grid; finally, the aspect ratio and DPI of the image were accurately calculated (250:135 and 101.6 DPI) to generate a clean, high-resolution PNG image (1000x540) without background borders or titles.

[0030] (5) Form a dataset of image and signal matching samples that are matched one-to-one with the standardized signals in ECG images and index files, i.e., high-quality image sample pairs of signals.

[0031] By precisely calculating the aspect ratio and DPI of the images, it is ensured that the rendered images visually simulate real ECG images. All generated images are saved as high-resolution PNG files in a clean format without background borders or titles, with the filenames associated with the corresponding patient and study IDs. This process creates a dataset containing matching (signal, image) pairs, forming the basis for multimodal training.

[0032] The multimodal self-supervised pre-training model building module is used to receive image sample pairs of signals, train the image encoder through multimodal contrastive learning, and output the trained ECG paper tape data base model.

[0033] The processing steps specifically include: (1) Model architecture design, forming an initial model architecture, including a frozen signal encoder, a trainable image encoder, and a projection head. As shown in Figure 2, the signal encoder ( As the "teacher" providing supervisory signals, a 12-lead ECG signal model—ECGFounder—was selected. This model had been trained and validated on 10 million data points in a large-scale classification task. Based on the one-dimensional convolutional network RegNet, it was deeply customized for the characteristics of ECG time-series data. The initial layer uses fewer channels to capture low-level features, while deeper layers gradually increase the number of channels and incorporate a bottleneck module. Combined with grouped convolution and channel attention mechanisms, it simultaneously strengthens the feature representation in both temporal and spatial dimensions, effectively extracting high-level physiological features from the original waveform.

[0034] Therefore, instead of fine-tuning the ECGFounder, it is used as a fixed, pre-trained feature encoder within this framework. Specifically, the 12-lead ECG signal (12x1250) after filtering, normalization, and length standardization is fed into the ECGFounder model. The model maps the original temporal signal to a high-dimensional latent space through its deep convolutional network. Finally, global average pooling is performed on the feature map output from the last convolutional stage to obtain a 1024-dimensional fixed-length feature vector representing the entire ECG record. This vector, as a high-quality representation of the signal modality, provides a stable supervisory signal for the training of the image modality and, under the action of the contrastive loss function, guides the image encoder to learn a visual representation with equivalent clinical semantics that aligns with it.

[0035] As shown in Figure 3, for the image encoder ( The image encoder was pre-trained using the VisionTransformer (ViT-B / 32) model from the OpenAI CLIIP project. Key structural adaptations were performed to enable it to process multi-lead ECG images. The original ViT's first convolutional layer (pattern bedding layer) was designed for processing 3-channel (RGB) images. This was replaced with a new convolutional layer with 12 input channels, and the weights were initialized by averaging the original RGB weights across the channel dimension and replicating them 12 times. This design allows the model to treat all 12 leads as 12 input channels, simultaneously processing and fusing the spatial information from all leads. During training, the parameters of this image encoder were one of the only parts that needed to be optimized via backpropagation. The network structure of the image encoder is shown in Figure 4.

[0036] Image projection head ( To compute the contrast loss in the same space, a simple linear layer was added as a projection head after the image encoder. Original features of the output A linear transformation is performed to a space consistent with the signal embedding dimension, resulting in the final image embedding used for comparison. .

[0037] (2) Based on multimodal contrastive learning, construct an ECG paper tape data base model.

[0038] As shown in Figure 5, contrastive learning is a powerful self-supervised learning method whose core idea is "learning through comparison." It doesn't directly predict a specific label, but instead learns an embedding space where semantically similar samples (positive sample pairs) are brought closer together, while semantically dissimilar samples (negative sample pairs) are pushed apart. In this study, this idea is innovatively applied to cross-modal ECG data. The aim is to utilize signal-image sample pairs to bring features from two modalities of the same ECG recording closer together in the feature space, while features from different recordings are pushed apart, achieving cross-modal feature comparison and joint learning to uncover common features and complementary information related to cardiomyopathy. Ultimately, the desired ECG tape data base model is obtained.

[0039] Contrast Learning Objective: We employ the InfoNCE (Noise Contrastive Estimation) loss function to drive model learning. For a training batch (mini-batch) containing N (signal, image) sample pairs, we first obtain N signal embeddings through their respective encoders and projectors. and N image embeddings For any signal embedding in the batch Its corresponding image embedding It is the only positive sample, while all other image embeddings within the batch are... All samples are negative. The loss function is designed to be symmetric, simultaneously optimizing the alignment in both the "signal-to-image" and "image-to-signal" directions: the loss function for image-to-signal... : signal-to-image loss : in, It is cosine similarity. It is a learnable temperature parameter that controls the sharpness of the softmax function and is crucial to the stability of the learning process. Total contrast loss. The total comparative loss is the average of the losses in these two directions, which is then summed and averaged over the entire batch. In the aforementioned comparative learning pre-training phase, we employed an asymmetric "teacher-student" training strategy. Specifically, the signal encoder ( All network parameters of the image encoder will be completely frozen and will not participate in backpropagation and gradient updates. ) and image projection head ( The parameters of ) are trainable.

[0040] Based on the above calculation and optimization process, the steps to obtain the final ECG paper tape data base model are as follows: (1) Determine the contrast learning scheme and target. The contrast learning self-supervised learning method is innovatively applied to ECG cross-modal data. By using signal-image sample pairs, the two modal features of the same ECG record are brought close together in the feature space, while the features of different records are far apart, so as to realize cross-modal feature comparison and joint learning, and to mine common features and complementary information related to cardiomyopathy. The InfoNCE loss function is used to drive model learning. For a training batch containing N (signal, image) sample pairs, the signal embedding and image embedding are obtained through their respective encoders and projectors. The image embedding corresponding to each signal embedding is set as a positive sample, and other image embeddings in the batch are set as negative samples. A symmetrical loss function is designed, and the alignment of the two directions of "signal to image" and "image to signal" is optimized at the same time. The loss from image to signal and the loss from signal to image are calculated, and the average of the two is taken to obtain the total contrast loss. The "teacher-student" asymmetric training strategy is adopted to freeze all network parameters of the signal encoder and train only the parameters of the image encoder and the image projector.

[0041] (2) Configuring Experimental and Training Parameters All experiments were conducted using a deep learning framework based on PyTorch (Python 3.9). The hardware environment was configured with a single NVIDIA RTX3090 GPU. The optimizer used was AdamW, and the weight decay was set to 0.05. Hyperparameters were set in stages. During the pre-training stage, the learning rate was set to 1e-5, the batch size was set to 32, the training period was 30, and the cumulative training time was approximately 200 hours. During the downstream fine-tuning stage, the classifier learning rate was set to 1e-3, and the batch size and training period remained unchanged at 32 / 30.

[0042] (3) Determine if training converges and terminate pre-training. When the total contrast loss L converges (the loss basically stops decreasing and tends to 0), and the cross-modal retrieval accuracy reaches a plateau on the validation set, the pre-training process is terminated.

[0043] (4) Model decoupling and instantiation: Remove the frozen signal encoder branch, which is only used to construct the feature space constraints for contrastive learning and is no longer needed in the inference stage; retain only the trained image encoder, which has learned ECG-related electrophysiological prior knowledge and can be used as a general visual backbone network. Through simple fine-tuning, it can be adapted to various downstream clinical diagnostic tasks to obtain the final ECG paper tape data base model.

[0044] This strategy is designed based on the following key considerations: (1) Providing a stable learning objective: The signal encoder is a pre-trained, high-performance model that can be regarded as a "domain expert" or "teacher". Freezing it means that it can provide a stable, high-quality feature representation for each signal sample. This provides a fixed and clear learning objective for the image encoder, which is the "student". If the two encoders are trained at the same time, the learning objective will keep drifting, which may lead to instability or difficulty in convergence during the training process.

[0045] (2) Focus on cross-modal knowledge transfer: This strategy places all the learning pressure on the image encoder. Its sole task is to adjust its own parameters so that its output feature representation can “mimic” and “align” with the signal representation provided by the teacher model. This makes the optimization process focus entirely on the core task of knowledge transfer from the signal modality to the image modality.

[0046] (3) Effectively prevent model collapse: In contrastive learning, a potential risk is that the model learns a "shortcut" solution, such as all outputs becoming constant, i.e., model collapse. Freezing a branch is a simple and effective way to prevent this phenomenon because it breaks the symmetry of the model, making it possible to learn a trivial solution without reaching the minimum of the loss function.

[0047] The intelligent image cropping module is used to convert real-world ECG scans into standardized input images, providing adapted input for image models without the need to retrain them.

[0048] To enable the pre-trained model to be directly applied to real-world scans without retraining, a standalone intelligent cropping module was developed. This module is implemented based on a series of classic computer vision techniques, including adaptive thresholding, morphological image processing, and contour analysis. These techniques are innovatively combined into a hybrid strategy-based intelligent cropping workflow that integrates traditional image processing logic with content-based adaptive judgment, aiming to robustly locate and extract standardized ECG core content from complex backgrounds.

[0049] Specifically, this includes: (1) Macro-Localization: Finding the largest content outline in the image through adaptive thresholding and morphological processing. This step aims to robustly identify the “paper” area containing all ECG information and perform preliminary cropping to remove large areas of invalid background borders.

[0050] (2) Fine-grained Hybrid Crop: The vertical axis sets a grid height cropping ratio, retaining only 78% of the initial cropped area from the top. This operation reliably removes the bottom strip leads and other footer information while fully preserving the 12-lead matrix above. For the horizontal axis, the darkest signal waveform is detected using a brightness threshold, and its maximum horizontal boundary is found. This adaptively locates the start and end points of the actual signal, achieving compact horizontal cropping and eliminating unnecessary blank margins on both sides.

[0051] (3) Standardization: Scale the final cropped image to a uniform size (1000x540 pixels). Ensure that all images input to the deep learning model are of the same size, which is a necessary prerequisite for batch training.

[0052] The following analysis presents the specific experimental process and results: This study uses three datasets: the MIMIC-IV-ECG dataset, the PTB-XL dataset, and the National Heart Foundation 2023ECG dataset from the Kaggle platform. The MIMIC-IV-ECG dataset is used for fine-tuning the ViT model during the pre-training phase, while the PTB-XL and National Heart Foundation 2023ECG datasets are used for downstream task evaluation.

[0053] The MIMIC-IV-ECG dataset, part of the MIMIC-IV project, contains 800,000 12-lead ECG records from Beth Israel Deaconess Medical Center in Boston, in WFDB format. Its advantage lies in its large scale and the availability of raw data; however, most records lack uniform, machine-readable diagnostic labels, making it suitable for validating self-supervised feature learning methods on unlabeled data. To ensure the quality of image model fine-tuning, 12,944 samples containing signal artifacts, partial lead loss, or electrode detachment leading to complete lead loss were removed.

[0054] The PTB-XL dataset contains 21,799 clinical 12-lead ECG records from 18,869 patients, covering a variety of cardiac pathological conditions. This dataset provides detailed, standardized, multi-label diagnostic information for each record, validated by cardiologists, and offers a dataset partitioning scheme based on patient ID. It serves as a fair and reproducible benchmark platform for evaluating and comparing the performance of different models.

[0055] The National Heart Foundation 2023 ECG dataset comes from the National Heart Foundation of Bangladesh and contains four categories of images: ECG images of patients with arrhythmias, patients with myocardial infarction, normal individuals, and patients with a history of myocardial infarction.

[0056] To prevent data leakage, the dataset is strictly partitioned by patient ID. Random partitioning could result in records from the same patient at different times being simultaneously assigned to both the training and test sets, causing the model to learn patient-specific physiological characteristics rather than generalized disease pathological patterns. Partitioning by patient ID ensures that the training, validation, and test sets are independent at the patient level, allowing the evaluation results to accurately reflect the model's generalization ability and avoiding overfitting to specific individual characteristics. The specific dataset partitioning is shown in the figure.

[0057] To evaluate the model's ability to establish accurate correspondences between two different data modalities—ECG signals and ECG images—a cross-modal retrieval task was designed. On the MIMIC-IV-ECG dataset test set, embedding vectors for all signal and image samples were pre-calculated. Using a signal embedding as the query, cosine similarity was calculated and ranked across all image embeddings, and vice versa. Recall@K (R@1, R@5, R@10) was used as the evaluation metric, representing the probability that a correct match appears in the Top-K retrieval results. A high R@K value directly demonstrates that the model successfully maps semantically equivalent signals and images to neighboring positions in the embedding space, providing direct evidence for the optimization of the contrastive learning objective. Table 1 shows that the accuracy is above 90% regardless of the strictest Recall-1 or the relaxed Recall-5 and Recall-10 criteria, even reaching 99% after relaxation. This indicates that after contrastive learning pre-training, the image model can understand that the signal and image depict the same cardiac event, even though their data formats are completely different. As shown in Figure 6, the downstream task employs a strict linear probing protocol: the pre-trained image encoder and image projector are completely frozen during the fine-tuning phase, and the newly added linear classifier is trained only on top of them. This method reflects the linear separability and quality of the pre-trained features themselves in the purest way. This method is compared with two key baseline models: (1) the ViT-B / 32 model with the same structure but trained from scratch to demonstrate the value of pre-training; and (2) the ViT model pre-trained on ImageNet to demonstrate that domain-specific self-supervised pre-training outperforms general-domain supervised pre-training.

[0058] The experimental results on the PTB-XL dataset are shown in Table 2 and Figure 7 below. It can be seen that both the ViT-B / 32 model trained from scratch in a supervised manner and the ViT model trained in a supervised manner on ImageNet-21k outperform our model, indicating that our pre-training is effective and can outperform supervised models in the general domain.

[0059] Experimental results on the National Heart Foundation 2023 ECG dataset are shown in Table 3 and Figure 8. It can be seen that the ViT model pre-trained on the large-scale ImageNet-21k dataset significantly outperforms the standard ViT-B / 32 baseline model (AUC between 0.95 and 0.99) (AUC between 0.91 and 0.98), demonstrating the effectiveness of general pre-training in improving ECG feature extraction capabilities.

[0060]

Claims

1. A system for constructing an ECG paper tape data base model based on multimodal contrastive learning, comprising three parts: an image sample pair signal generation module, a multimodal self-supervised pre-training model construction module, and an intelligent image cropping module; the image sample pair signal generation module converts the original ECG signal into a standardized signal and matching image, outputs the image sample pair signal of the sample pair dataset, and inputs it into the multimodal self-supervised pre-training model construction module; the multimodal self-supervised pre-training model construction module receives the image sample pair signal, trains the image encoder through multimodal contrastive learning, and outputs the trained ECG paper tape data base model; the intelligent image cropping module converts real-scene ECG scans into standardized input images, providing adapted input for the image model without retraining the model.

2. The model building system according to claim 1, characterized in that, The image sample pair signal generation module performs a standardized preprocessing procedure on the original one-dimensional ECG signal to remove baseline drift and interference noise.

3. The model building system according to claim 2, characterized in that, The removal of baseline drift and interference noise specifically includes: (1) Filtering: Applying a 4th-order Butterworth bandpass filter to remove baseline drift and high-frequency noise while retaining the key components of the ECG waveform; (2) Length normalization: All signals are truncated or padded to a fixed length; (3) Amplitude normalization: Z-score normalization is performed on each of the 12 leads; (4) Image rendering: First, a plotting module is customized based on the open-source ecg_plot library, and the image is rendered in accordance with the clinical ECG display standard. Then, the 12 leads are arranged in a grid layout of 4 columns and 3 rows, with a background containing a standard red millimeter grid. Finally, the aspect ratio and DPI of the image are accurately calculated to generate a clean, high-resolution PNG image without background borders or titles; (5) A dataset of image and signal matching pairs is formed, which is a high-quality image sample pair signal, that is, a dataset of image and signal matching pairs that are matched one-to-one with the normalized signals.

4. The model building system according to claim 1, characterized in that, The specific processing steps of the multimodal self-supervised pre-training model construction module include: (1) Model architecture design, forming an initial model architecture, including a frozen signal encoder, a trainable image encoder and an image projection head; (2) Based on multimodal contrastive learning, constructing an ECG paper tape data base model.

5. The model building system according to claim 1, characterized in that, The intelligent image cropping module is based on a series of classic computer vision techniques, including adaptive thresholding, morphological image processing and contour analysis, and combines these techniques into an intelligent cropping process based on a hybrid strategy.

6. The model building system according to claim 5, characterized in that, The intelligent cropping process based on the hybrid strategy specifically includes: (1) Macro content positioning: Find the content outline with the largest area in the image through adaptive threshold and morphological processing; (2) Refined hybrid cropping: Set a grid height cropping ratio on the vertical axis and retain only 78% of the height of the initial cropping area starting from the top; (3) Size standardization: Scale the final cropped image to a uniform size.

7. The model building system according to claim 4, characterized in that, The image projection head is designed to compute contrast loss in the same space. A simple linear layer is added after the image encoder to linearly transform the original output features to a space consistent with the signal embedding dimension, resulting in the final image embedding used for contrast.

8. The model building system according to claim 4, characterized in that, The construction of the ECG paper tape data base model based on multimodal contrastive learning is as follows: (1) Determine the contrastive learning scheme and objectives; (2) Configure experimental and training parameters; (3) Determine the training convergence and terminate the pre-training; (4) Decouple and instantiate the model.

9. The model building system according to claim 8, characterized in that, The process of converting real-world ECG scans into standardized input images specifically includes: obtaining signal embeddings and image embeddings through their respective encoders and projectors; setting the image embedding corresponding to each signal embedding as a positive sample and other image embeddings within the batch as negative samples; designing a symmetrical loss function; simultaneously optimizing the alignment in both the "signal to image" and "image to signal" directions; calculating the loss from image to signal and the loss from signal to image; and then taking the average of the two to obtain the total contrast loss.

10. The model building system according to claim 9, characterized in that, Image-to-signal loss ： Signal-to-image loss ： ;in, It is cosine similarity. It is a learnable temperature parameter that controls the sharpness of the softmax function and is crucial to the stability of the learning process. Total contrast loss. The total comparative loss is the average of the losses in these two directions, which is then summed and averaged over the entire batch. 。

Images