Multimodal electrocardiogram diagnosis large model architecture and training method

Abstract

This invention discloses a multimodal ECG diagnostic large-scale model architecture, belonging to the field of machine learning technology, to solve the problem of inaccurate text-image understanding of existing ECG data. The architecture includes: an ECG encoder, an image encoder, a feature alignment module, and a large-scale language model. The ECG encoder encodes the input ECG time-series signal into a first feature representation; the image encoder encodes the input ECG image into a second feature representation; and the feature alignment module connects the ECG encoder and the image encoder. This invention uses contrastive learning to enable the model to understand the relationship between signals and text, ensuring the medical accuracy of the generated content.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and more particularly to a large-scale model architecture and training method for electrocardiogram diagnosis based on multimodality. Background Technology

[0002] Electrocardiogram (ECG) is an important tool for diagnosing heart disease. With the development of artificial intelligence technology, using machine learning algorithms to assist in the analysis of ECG data has become a research hotspot. However, existing technical solutions still have limitations in practical applications:

[0003] First, traditional ECG AI diagnostic algorithms typically treat ECG signals as single time-series data and are limited to classification tasks (such as arrhythmia classification). These methods cannot generate structured diagnostic reports that patients or doctors can directly understand, nor can they answer clinical questions related to specific cases, still requiring secondary interpretation by doctors, resulting in low levels of intelligence and diagnostic efficiency.

[0004] Secondly, although the general multimodal large language model (MLLM) is strong in text and image understanding and generation, it suffers from a serious "illusion" problem (factual error) when directly applied to specialized fields such as cardiology. That is, the model may generate medical descriptions or diagnoses that seem reasonable but are factually incorrect, resulting in insufficient reliability of its analysis results and making it unsuitable for direct application in serious clinical auxiliary diagnostic scenarios.

[0005] Furthermore, most publicly available ECG datasets lack comprehensive, detailed, and formatted text descriptions, exhibiting highly structured text and repetitive terminology. This fragmentation between data modalities makes it difficult for models to learn the deep connections between ECG signals and rich medical semantics, hindering the development of cross-modal understanding and report generation capabilities.

[0006] Therefore, there is an urgent need for an intelligent diagnostic solution that can accurately understand electrocardiogram signals, generate reliable diagnostic reports, and has professional question-and-answer capabilities, in order to overcome the shortcomings of existing technologies, such as limited functionality, poor reliability, and insufficient data utilization. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, one of the objectives of this invention is to provide a large-scale ECG diagnostic model architecture based on multimodal learning, which combines contrastive learning with signal and image fusion to construct a large-scale ECG diagnostic model architecture to generate accurate medical reports.

[0008] One of the objectives of this invention is achieved through the following technical solution: A large-scale ECG diagnostic model architecture based on multimodality includes: an ECG encoder, an image encoder, a feature alignment module, and a large-scale language model; The ECG encoder is used to encode the input ECG time-series signal into a first feature representation. The ECG encoder is based on a one-dimensional Vision Transformer architecture and is trained through a contrastive learning framework to map the ECG signal to a shared semantic space aligned with the medical text description. The image encoder is used to encode the input electrocardiogram image into a second feature representation; The feature alignment module is used to connect the ECG encoder and the image encoder, and project the first feature representation and the second feature representation onto a unified feature space to obtain fused multimodal features; A large language model, connected to the feature alignment module, is used to take the fused multimodal features as input and output diagnostic reports or Q&A content related to ECG data.

[0009] By constructing a system architecture that includes an ECG encoder, an image encoder, a feature alignment module, and a large language model, the system systematically achieves the fusion of dual-modal information from ECG time-series signals and ECG images, providing a hardware foundation for a comprehensive and accurate understanding of ECG data.

[0010] To achieve efficient and information-rich feature representation, the ECG encoder includes the following components connected in sequence: The slice embedding layer is used to segment multi-lead ECG timing signals into multiple non-overlapping segments through one-dimensional convolution operations and convert them into a sequence of embedding vectors. A position and category embedding layer is used to add a learnable classification token as the first element of the sequence at the front of the embedding vector sequence, and to add a learnable position encoding for each classification token in the sequence; A multi-layer Transformer encoder is used to extract and aggregate global contextual features from a classification token sequence after adding positional encoding through a multi-head self-attention mechanism. The pooling layer is used to extract the vector corresponding to the classification token after processing by the multi-layer Transformer encoder, as the first feature representation.

[0011] Furthermore, in order to enable the model to understand signal timing and global semantics, the stride of the one-dimensional convolutional layer is equal to its kernel size, so as to convert the ECG timing signal into a non-overlapping sequence of segments; the number of input channels of the one-dimensional convolution operation corresponds to the number of leads in the ECG. The location and category embedding layer is configured to perform: Concatenation operation: A learnable classification token vector is concatenated to the beginning position of the embedded vector sequence to form an extended sequence; Position encoding operation: Add an element-wise a learnable one-dimensional position encoding matrix to the extended sequence, wherein each row of the position encoding matrix corresponds to a position encoding vector at a specific position in the extended sequence; The multilayer Transformer encoder consists of multiple stacked residual attention blocks, each of which includes a layer normalization module, a multi-head self-attention mechanism module, residual connections, and a feedforward neural network.

[0012] To achieve seamless integration across modal semantic spaces, the ECG encoder also includes a projection head connected to the pooling layer, used to map the first feature representation to a preset shared embedding space dimension.

[0013] Furthermore, to improve training efficiency and feature quality, the ECG time-series signal is converted into the input ECG image using an ECG-image-kit; the image encoder is a pre-trained visual encoder based on the Vision Transformer architecture.

[0014] The feature alignment module maps and merges the dimensions of the first feature representation and the second feature representation into a unified feature dimension; the feature alignment module is a multilayer perceptron, which is a two-layer perceptron containing two linear layers and a GELU activation function; during training, the parameters of the ECG encoder and the image encoder are frozen, and the parameters of the feature alignment module and the parameters of the large language model are jointly trained.

[0015] During the fine-tuning phase, the parameters of the pre-trained ECG and image encoders are frozen, and only the feature alignment module and the large language model are trained together. This approach not only protects the valuable features already learned by the encoder from being destroyed, but also allows the fusion module and LLM to quickly adapt to downstream tasks, preventing overfitting and significantly improving training stability and final performance.

[0016] To achieve flexible, controllable, and task-oriented intelligent interaction, the large language model receives fused feature representations from the feature alignment module and text instructions from external input, and generates structured text output related to electrocardiogram (ECG) diagnosis. The text instructions are text requests used to guide and limit the model to perform specific tasks. The text instructions from external input include at least one of the following types: instructions to request the generation of ECG diagnostic reports, instructions to ask questions to interpret ECG abnormalities, and instructions to request the interpretation of ECG features.

[0017] The second objective of this invention is to provide a training method for a large-scale ECG diagnostic model based on multimodal approaches.

[0018] The second objective of this invention is achieved by the following technical solution: A training method for a large-scale ECG diagnostic model architecture based on multimodality, used to train the large-scale model architecture as described in any one of claims 1-7, comprising: Using a contrastive learning framework, the original ECG time-series signal and its corresponding medical text description are used as positive sample pairs for training. The ECG encoder is optimized so that its output first feature representation is aligned with the semantic representation of the medical text description in a shared embedding space. The original ECG timing signal is input into the pre-trained ECG encoder to obtain a first feature representation; the ECG image is input into the image encoder to obtain a second feature representation. The first feature representation and the second feature representation are fused and dimensionally aligned to generate a unified fused feature representation.

[0019] Furthermore, co-evolution of understanding and generation is achieved through joint optimization. The contrastive learning framework is the COCA framework, and its training objective is to simultaneously minimize the contrastive loss and the text description generation loss.

[0020] Furthermore, the accuracy of cross-modal mapping is ensured through bidirectional alignment. The contrast loss includes signal-to-text contrast loss and text-to-signal contrast loss, which are calculated using a normalized exponential function and cross-entropy loss. The contrast loss is the mean of the sum of the signal-to-text contrast loss and the text-to-signal contrast loss. The signal-to-text comparison loss is used to maximize the similarity between ECG signal features and corresponding text features, while minimizing the similarity between the ECG signal features and non-corresponding text features. The text-to-signal contrast loss is used to maximize the similarity between text features and corresponding ECG signal features, while minimizing the similarity between the text features and non-corresponding ECG signal features.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention enables the model to understand the relationship between signals and text through contrastive learning, integrating the dual information of signals and images. This allows the model to not only analyze raw ECG waveform signals but also interpret ECG images. By combining an ECG encoder trained through contrastive learning and possessing semantic understanding capabilities with a large-scale language model with powerful natural language generation capabilities, the system can not only classify signals but also automatically generate structured diagnostic reports and provide professional Q&A, achieving optimization from "aiding classification" to "aiding diagnosis and interpretation." The ECG encoder, trained through a contrastive learning framework, aligns its output features with medical text in the semantic space, injecting prior medical knowledge logic into the model. This effectively alleviates the "illusion" problem that general-purpose large models easily produce in vertical domains, ensuring the medical accuracy of the generated content. Attached Figure Description

[0022] Figure 1 This invention provides a flowchart and structure diagram of a large-scale ECG diagnostic model based on multimodal operation. Figure 2 This is a schematic diagram of the architecture of the large-scale ECG diagnostic model based on multimodal methods in Example 1; Figure 3 This is a schematic diagram of the feature alignment process in Example 1. Detailed Implementation

[0023] The present invention will now be described in more detail with reference to the accompanying drawings. It should be noted that the following description of the present invention with reference to the accompanying drawings is merely illustrative and not restrictive. Various embodiments can be combined with each other to form other embodiments not shown in the following description.

[0024] Example 1 Example 1 provides a large-scale ECG diagnostic model architecture based on multimodality, which aims to construct a joint model by simultaneously fusing raw ECG time-series signals and visualized images, and to perform in-depth fine-tuning in the ECG vertical domain.

[0025] Please refer to Figure 1 and Figure 2 The diagram shows the architecture and processing flow. The WFDB (a tool for processing physiological signals) shown in the diagram is a preliminary step in this architecture, which is subsequently converted into the token described in this embodiment by the encoder. WFDB is a conventional technique and will not be elaborated upon here. This embodiment's multimodal ECG diagnostic large-scale model architecture includes: an ECG encoder, an image encoder, a feature alignment module, and a large language model. The ECG encoder is used to encode the input ECG time-series signal into a first feature representation. The ECG encoder is based on a one-dimensional Vision Transformer architecture and is trained through a contrastive learning framework to map the ECG signal to a shared semantic space aligned with the medical text description. In this embodiment, the ECG encoder uses the ECG-Encoder architecture, which is a Transformer-based ECG encoder architecture. Its core design idea is to process the ECG signal into a sequence of "visual image patches", which is essentially a variant of Vision Transformer (ViT) on 1D time-series signals.

[0026] The design principle of the ECG encoder lies in avoiding directly inputting the original 5000 sampling points into the Transformer to prevent excessive computation. Instead, it segments the ECG signal into patches using convolutional layers. This not only reduces the sequence length but also extracts local features (such as QRS complex morphology). Furthermore, the architecture in this embodiment utilizes the Transformer's self-attention mechanism for global context processing, allowing the model to capture relationships between distant features in the ECG, such as the interval between P waves and QRS waves, and global patterns of arrhythmias. It also borrows from BERT / ViT, using a learnable special token to aggregate the global semantic information of the entire ECG for subsequent alignment or classification.

[0027] Specifically, assuming the input data dimension is: [Batch, Lead=12, Length=5000], the ECG encoder includes the following components connected in sequence: The slice embedding layer is used to segment multi-lead ECG timing signals into multiple non-overlapping segments through one-dimensional convolution operations and convert them into a sequence of embedding vectors. The Patch Embedding layer is a layer that uses a one-dimensional convolution operation (nn.Conv1d) to convert the original electrocardiogram (ECG) signal into a series of embedding vectors (token sequences), which serve as the basic units for subsequent model processing.

[0028] Specifically, the stride of a one-dimensional convolutional layer is equal to its kernel size to convert ECG time-series signals into non-overlapping segment sequences. The number of input channels for the one-dimensional convolution operation corresponds to the number of ECG leads. Typically, the input signal has 12 channels, corresponding to a standard 12-lead ECG with a length of 5000 time points; however, the number of channels can be determined based on the actual number of ECG leads. This layer uses a one-dimensional convolutional kernel with a kernel size and stride of patchsize=50 to perform non-overlapping segmentation of the signal in each lead. Since the stride is equal to the kernel size, each slide skips exactly one complete window, thus dividing the original signal into non-overlapping "segments," or patches.

[0029] In this embodiment, each 50-length segment is mapped to a width-dimensional embedding vector through a convolution operation. Since the total input length is 5000, it can be divided into 5000 / 50 = 100 such segments, resulting in 100 tokens. The output tensor of the convolution operation has a shape of [Batch, Width, 100], which is then adjusted to [Batch, 100, Width] through dimensionality transpose to meet the input format requirements of sequence models such as Transformer, i.e., [batch size, sequence length, feature dimension].

[0030] The above-mentioned slice embedding layer process realizes the transformation from the original continuous time-series signal to the discrete token sequence, which is the first step for the model to understand the electrocardiogram structural information.

[0031] A position and category embedding layer is used to add a learnable classification token as the first element of the sequence at the front of the embedding vector sequence, and to add a learnable position encoding for each classification token in the sequence; The Position & Class Embedding layer is configured to perform: Concatenation operation: A learnable classification token vector is concatenated to the beginning position of the embedded vector sequence to form an extended sequence; Position encoding operation: Add an element-wise a learnable one-dimensional position encoding matrix to the extended sequence, wherein each row of the position encoding matrix corresponds to a position encoding vector at a specific position in the extended sequence; Specifically, after converting the original ECG signal into a sequence of 100 tokens, in order to adapt to the input requirements of the Transformer architecture, this embodiment introduces two key learnable embedding components into the model: class embedding and positional embedding.

[0032] The class embedding described above defines a learnable vector: `(self.class_embedding)`, whose dimension matches the feature dimension (width) of the token. This vector, as a special [CLS] token, is concatenated to the beginning of the ECG token sequence. After concatenation, the sequence length is expanded from 100 to 101. This [CLS] token aggregates global information from the entire sequence during training and is typically used for downstream tasks, such as the final prediction output of classification.

[0033] Since the Transformer model itself lacks the ability to perceive the input order (i.e., it is "position-insensitive"), positional information must be explicitly injected. Therefore, this embodiment constructs the aforementioned positional embedding. Specifically, a learnable positional encoding matrix is ​​introduced: (self.positional_embedding), with a shape of [0101, width], corresponding to a unique embedding vector for each position in the sequence (including position 0 of the [CLS] token and positions 1~100 of the subsequent 100 ECG tokens). This positional embedding is added element-wise to the concatenated token sequence, enabling the model to distinguish the chronological order of different time segments in the original ECG signal, preserving temporal structure information.

[0034] After processing in this layer, the input sequence becomes a tensor of shape [Batch, 101, Width], which contains both content information (ECG patches + [CLS]Token) and positional context, preparing it for subsequent processing by the Transformer encoder.

[0035] A multi-layer Transformer encoder is used to extract and aggregate global contextual features from a classification token sequence after adding positional encoding through a multi-head self-attention mechanism. It consists of multiple stacked residual attention blocks, each of which includes a layer normalization module, a multi-head self-attention mechanism module, residual connections, and a feedforward neural network.

[0036] The multi-layer Transformer encoder, implemented by self.transformer, is responsible for deep feature extraction and contextual information aggregation of the embedded token sequence. It is based on the standard Transformer encoder architecture and consists of multiple stacked residual attention blocks, with a default configuration of 12 layers, each with the same structure.

[0037] Each coded block contains the following key components, which are executed in sequence: First, Layer Normalization (LN) is performed. Before entering the attention mechanism, the input token sequence is normalized along the feature dimension to stabilize the training process, accelerate convergence, and reduce sensitivity to initialization and learning rate.

[0038] Then, the Multi-Head Self-Attention (MSA) mechanism is executed, which is also the core mechanism of the Transformer. Each token (including the initial [CLS] token and 100 ECG tokens) interacts with all other tokens, dynamically weighting and aggregating global information by calculating attention weights. By default, 12 attention heads are used, each learning a different attention pattern independently. Finally, the results are concatenated and linearly transformed back to the original dimension (width=768).

[0039] In electrocardiogram (ECG) analysis tasks, this mechanism enables models to capture local morphological features within the cardiac cycle (such as P waves, QRS complexes, and T waves) as well as rhythmic dependencies across time steps (such as RR interval changes), thereby understanding the temporal structure and pathological patterns of ECG signals.

[0040] Residual connections are formed by adding the self-attention output to the input. They can alleviate the vanishing gradient problem in deep networks, allowing the model to be trained stably on deeper structures (such as 12 layers).

[0041] The result after residual connections is normalized again and then fed into a two-layer fully connected feed-forward network (MLP / Feed-Forward Network). This typically involves first increasing the dimensionality and then decreasing it, introducing nonlinear transformation capabilities to further enhance the model's expressive power. This MLP also incorporates residual connections.

[0042] The steps described above constitute a complete Transformer encoding block. The entire encoder repeats this block 12 times, refining higher-level semantic representations layer by layer. As the number of layers increases, the [CLS] Token gradually integrates global contextual information from the entire ECG sequence, providing a strong feature foundation for subsequent classification or diagnostic tasks.

[0043] Finally, the layer outputs a tensor of shape [Batch, 101, 768], where the 0th position (i.e., the [CLS]Token) is typically used as the prediction head input for downstream tasks.

[0044] The pooling layer (LayerNorm & Pooling) is used to extract the vector corresponding to the classification token after processing by the multi-layer Transformer encoder, as the first feature representation.

[0045] After the Transformer model encoder outputs a high-dimensional sequence representation, a pooling layer is used to generate the final ECG feature vector for downstream tasks such as classification or multimodal alignment.

[0046] Specifically, the pooling layer includes post-layer normalization and global pooling. Post-layer normalization applies layer normalization once to the entire token sequence output by the Transformer, normalizing each token of each sample along the feature dimension (e.g., 768 dimensions). This step is part of the "Post-Norm" strategy, used to further stabilize the feature distribution, improve the model's generalization ability, and provide more normalized input for subsequent projection or loss calculations. Global pooling, to compress the 101-token sequence into a fixed-length global representation, uses the default pooling strategy "tok" (i.e., token pooling). This strategy selects only the first token in the sequence—the [CLS] token—as the feature summary of the entire ECG signal. This selection is based on the following design principle: during training, the [CLS] token continuously interacts with all other ECG tokens through a multi-layer self-attention mechanism, gradually aggregating the global context information of the entire 5000 ECG signals. Therefore, its final state can be regarded as a highly abstract, compact, and semantically rich representation of the entire ECG.

[0047] After this layer of processing, the original [Batch, 101, Width] sequence is transformed into a feature matrix of shape [Batch, Width], where each row corresponds to the global ECG embedding of a sample, which can be directly used for subsequent linear classification heads, contrastive learning projections, or other diagnostic tasks.

[0048] In addition, the ECG encoder in this embodiment also includes a projection head connected to the pooling layer, used to map the first feature representation to a preset shared embedding space dimension.

[0049] Specifically, after obtaining the global ECG features (usually 768-dimensional) after layer normalization and pooling, the model introduces a learnable linear projection matrix, i.e., a projection head, to map the ECG features into a shared semantic embedding space of a preset dimension.

[0050] The core function of this layer is cross-modal dimensional alignment. Through this linear transformation, the high-dimensional features of the ECG modality are compressed or expanded to the same dimensional space as the text encoder output vector. This alignment allows samples from different modalities (ECG signals and natural language descriptions) to be semantically compared in the same vector space.

[0051] In summary, the ECG-ENCODER is a multimodal ECG model that combines an ECG signal encoder (Transformer), a text encoder (such as BERT), a contrastive learning mechanism, and a generative text description module. It achieves cross-modal retrieval: inputting an ECG → finding a matching diagnostic report; or inputting a report → finding the corresponding ECG; and automatic text generation: generating natural language descriptions from ECGs, such as "sinus rhythm, ST segment depression".

[0052] The image encoder is used to encode the input electrocardiogram image into a second feature representation; The image encoder converts the ECG timing signal into the input ECG image using the ECG-image-kit; the image encoder is a pre-trained visual encoder based on the Vision Transformer architecture. The ECG-image-kit is a commonly used image processing tool, and its principles and processing procedures will not be elaborated upon in this embodiment.

[0053] The feature alignment module is used to connect the ECG encoder and the image encoder, and project the first feature representation and the second feature representation onto a unified feature space to obtain fused multimodal features; Please refer to Figure 3 The feature alignment process shown in the diagram includes the following steps: Collect case data from 1 million patients, including diagnostic classification, diagnostic details, electrocardiogram images, and 10-second electrocardiograms (TS) at 500 Hz; The ECG-image-kit is used to convert the temporal features of ECG into images, and image, temporal, and text classification datasets are constructed. Construct an ECG instruction dataset that includes both diagnostic and dialogue formats for instruction-based ECG LLM adjustment; Perform full fine-tuning on the LLM to enable it to understand ECG-related instructions.

[0054] The feature alignment module maps and merges the dimensions of the first feature representation and the second feature representation into a unified feature dimension; the feature alignment module is a multilayer perceptron, which is a two-layer perceptron containing two linear layers and a GELU activation function; during training, the parameters of the ECG encoder and the image encoder are frozen, and the parameters of the feature alignment module and the parameters of the large language model are jointly trained.

[0055] Specifically, the ECG time series is represented as , C It is the number of leads. L It is the length of the signal, that is, the number of time points (sampling points) for each lead. A ts This represents the original input matrix of the ECG timing sequence. Represents a two-dimensional tensor over the real number field, with the shape [lead number, time step].

[0056] The image time series is represented as , H and W These represent the image's height and width, respectively, with 3 indicating the RGB dimensions. B img Represents the image input tensor. It is a standard image format.

[0057] The feature vectors in the EC time series are obtained through the time encoder, and the pattern is as follows: ,in n s It is the number of time-series feature blocks. d s These are the dimensions of the temporal feature sequence, which are 50 and 768 in this embodiment, respectively. Visual information is processed by a pre-trained VISION-CLIT image encoder to extract features from the image, as shown in the following pattern: ,in n m It is the number of image feature blocks. d m It is the dimension of image features. e ts This represents the feature sequence after ECG encoding. E ts Indicates ECG timing encoder, Indicates encoder parameters, e img This represents the feature sequence after image encoding. E img This represents a pre-trained visual encoder. This represents the parameters of the visual encoder.

[0058] Using the outputs of the temporal encoder and spatial encoder as inputs to the LLM requires converting the input dimensions to a unified dimension. This embodiment also designs a two-layer multilayer perceptron (MLPX2_GELU) for the conversion, training only the linear projection layer.

[0059] Fine-tuning the LLM using instruction data: Freezing the temporal encoder and image encoder while jointly training the MLP alignment layer and the large model. The training objective is to minimize the negative log-likelihood of the target answer y and the large model response. , in, Represents the total loss function. x This represents the input features, specifically the first and second feature representations. Indicates factual labels, such as diagnostic conclusions and question-and-answer responses. For large language model parameters, N The sequence length is the number of tokens.

[0060] A large language model, connected to the feature alignment module, is used to take the fused multimodal features as input and output diagnostic reports or Q&A content related to ECG data.

[0061] In this embodiment, the large language model selected is the LLM large model (Qwen2.5). Of course, other large language models such as ChatGLM and Bloom can also be selected.

[0062] In summary, the advantages of the model architecture in this embodiment are: It simultaneously integrates the original ECG timing signal (TS) and the visualized image (IMG); it combines the intuitive waveform information provided by the image with the precise time characteristics provided by the timing sequence. This achieves the dual capabilities of "interpreting electricity from images" and "writing text based on electricity readings".

[0063] This architecture can support a complete "intelligent diagnosis and treatment closed loop" from "ECG input → automatic diagnosis → report generation → Q&A interaction".

[0064] Example 2 Example 2 is a training method based on the architecture of Example 1.

[0065] Specifically, training methods for a large-scale ECG diagnostic model architecture based on multimodal approaches include: Using a contrastive learning framework, the original ECG time-series signal and its corresponding medical text description are used as positive sample pairs for training. The ECG encoder is optimized so that its output first feature representation is aligned with the semantic representation of the medical text description in a shared embedding space. The aforementioned contrastive learning framework is the COCA framework, and its training objective is to simultaneously minimize both the contrastive loss and the text description generation loss. Specifically, it includes: Data preparation: Waveform signal data is parsed using WFDB, and waveforms are generated through feature calculation. Waveform enhancement datasets are then used as additional inputs to the waveform report to align ECG waveform data with the document report.

[0066] Contrastive learning: By maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs through contrastive learning (Con), electrocardiogram signals and text are mapped into a unified semantic space.

[0067] ECG time-series data is converted into feature representations, and contrast loss and descriptive loss are calculated. The contrast loss includes signal-to-text contrast loss and text-to-signal contrast loss, which are calculated using a normalized exponential function and cross-entropy loss. The contrast loss is the mean of the sum of the signal-to-text contrast loss and the text-to-signal contrast loss; where: The signal-to-text comparison loss is used to maximize the similarity between ECG signal features and corresponding text features, while minimizing the similarity between the ECG signal features and non-corresponding text features. The text-to-signal contrast loss is used to maximize the similarity between text features and corresponding ECG signal features, while minimizing the similarity between the text features and non-corresponding ECG signal features.

[0068] The original ECG timing signal is input into the pre-trained ECG encoder to obtain a first feature representation; the ECG image is input into the image encoder to obtain a second feature representation. The first feature representation and the second feature representation are fused and dimensionally aligned to generate a unified fused feature representation.

[0069] Specifically, the feature space of the ECG encoder is aligned with a large language model through a two-layer MLP adapter, and its computation satisfies: , , in, L e2t This represents the contrast loss from ECG to text. L t2e This represents the contrast loss from text to ECG. x i Indicates the first i The embedded representation of the central electrical timing signal data in the pair, i.e., the vector extracted from the ECG signal by the encoder. y i Indicates the first iThe feature representation of the corresponding text feature data is x i Paired positive samples, y j Represents all non-first i The obtained text feature data, i.e. the text embedding as negative samples, is used to calculate the interference terms (mismatched combinations). This represents a temperature parameter, usually a hyperparameter. The goal is to improve the matching score between the text and its corresponding ECG.

[0070] Then con-comparison loss , N This represents the batch size, the number of samples in the current batch, and the overall contrastive loss ensures alignment in both directions through addition. 1 / N This means averaging the loss to avoid the batch size affecting the absolute value of the loss, thus transforming the maximization objective into a minimization problem.

[0071] In addition, it also includes CA-described loss. : in, T Indicates the length of the text sequence. y t Indicates the first t Text tokens generated at each time step. y < t Indicates the previous generation t -1 word, This indicates the probability of predicting the next word. This represents the descriptive loss of text generation, i.e., the language modeling loss.

[0072] This loss represents the generation of high-quality textual descriptions (such as diagnostic conclusions or waveform analysis) from ECG features. It is a standard language modeling loss, where the total loss is the sum of all time steps, which is equivalent to maximizing the generation probability of the entire sentence to achieve the mapping from physiological signals to natural language.

[0073] Example 3 Example 3 illustrates the experimental results of the model architecture, training, and processing methods of Examples 1 and 2.

[0074] As shown in Table 1 below, by integrating time series data and electrocardiogram images through the aforementioned encoding model and unified architecture, as well as the model trained on the constructed heartbeat level interpretability dataset, the results are better than those of closed-source models such as GPT-4o and Claude3.5, and open-source models such as qwen2-vl and DeepSeek-VL.

[0075] Table 1

[0076] As shown in Table 1.1, the experimental evaluation in this embodiment covers 7 mainstream ECG datasets and 6 key metrics to comprehensively measure the model's performance in the following aspects: AUC, which is ranking ability and discrimination, the higher the better; F1, which is the balance between precision and recall, the higher the better; HL, which is the multi-label prediction error rate, the lower the better.

[0077] Table 1.1

[0078] The dataset is shown in Table 2.

[0079] Table 2

[0080] As can be seen from the above comparison, our model architecture achieves an AUC of 83.9 on the PTB-XL Super dataset, significantly outperforming the current best non-open-source model GPT-4o by 55.6 points, a gain of +28.3 points. This means that our model architecture has stronger discriminative power in complex classification tasks such as arrhythmia identification.

[0081] In clinical practice, a high AUC indicates that the system can more accurately separate abnormal ECGs from normal samples, reducing the risk of missed diagnoses.

[0082] On the CPSC2018 dataset, the F1 score reached 56.7, an improvement of +45 points compared to the best non-open-source model (Claude 3.5 Sonnet), while the original F1 score was only 11.5. This architecture achieves an F1 score of 56.7, a more than 4-fold improvement, demonstrating the model's strong robustness and generalization ability when handling multi-class and imbalanced label distributions. It is particularly suitable for tasks requiring simultaneous detection of multiple arrhythmias (such as atrial fibrillation, ventricular tachycardia, and premature beats). It solves the problem of high false positives and high false negatives in traditional models for fine-grained classification tasks.

[0083] The model achieved a score of 74.39 on the PTB-XL Report dataset, far exceeding the 50.2 improvement and 24.2 points of GPT-4o. This indicates that the model can not only "understand" ECG, but also generate high-quality, well-structured, and accurately terminological natural language reports, supporting the automated writing of preliminary diagnostic opinions and greatly reducing the burden on doctors.

[0084] On the CSN dataset, the accuracy reached 87.96%, surpassing all comparison models. It exceeded Qwen2-VL-72B's 35.5% and LLaVa-OneVision-72B's 44%, an improvement of more than 40 percentage points, indicating that the model has excellent performance in the recognition of specific pathological patterns (such as coronary artery disease and conduction block).

[0085] Furthermore, the accuracy reached 81.29% on the G12 dataset, significantly outperforming GPT-4o, indicating that the model possesses strong robustness against interference and contextual reasoning capabilities, making it suitable for rapid screening in the emergency department and assisting in determining whether immediate intervention is necessary. It achieved optimal results on the HL (multi-label prediction error rate) metric; on PTB-XL Super, HL = 10.8 (lower is better), indicating that the model is more accurate in handling complex diagnoses (such as simultaneous atrial fibrillation and ST-segment changes), reducing mislabeling and omissions, and avoiding treatment deviations due to labeling errors.

[0086] This embodiment also includes a horizontal comparison, please refer to Table 3.

[0087] Table 3

[0088] In summary, the model surpasses existing models in almost all metrics, especially in F1, accuracy, and report generation quality.

[0089] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

Claims

1. A large-scale ECG diagnostic model architecture based on multimodal approaches, characterized in that, include: ECG encoder, image encoder, feature alignment module, and large language model; The ECG encoder is used to encode the input ECG time-series signal into a first feature representation. The ECG encoder is based on a one-dimensional Vision Transformer architecture and is trained through a contrastive learning framework to map the ECG signal to a shared semantic space aligned with the medical text description. The image encoder is used to encode the input electrocardiogram image into a second feature representation; The feature alignment module is used to connect the ECG encoder and the image encoder, and project the first feature representation and the second feature representation onto a unified feature space to obtain fused multimodal features; A large language model, connected to the feature alignment module, is used to take the fused multimodal features as input and output diagnostic reports or Q&A content related to ECG data.

2. The large-scale ECG diagnostic model architecture based on multimodal approaches as described in claim 1, characterized in that, The electrocardiogram encoder comprises, in sequence, the following: The slice embedding layer is used to segment multi-lead ECG timing signals into multiple non-overlapping segments through one-dimensional convolution operations and convert them into a sequence of embedding vectors. A position and category embedding layer is used to add a learnable classification token as the first element of the sequence at the front of the embedding vector sequence, and to add a learnable position encoding for each classification token in the sequence; A multi-layer Transformer encoder is used to extract and aggregate global contextual features from a classification token sequence after adding positional encoding through a multi-head self-attention mechanism. The pooling layer is used to extract the vector corresponding to the classification token after processing by the multi-layer Transformer encoder, as the first feature representation.

3. The large-scale ECG diagnostic model architecture based on multimodal methods as described in claim 2, characterized in that, The stride of the one-dimensional convolutional layer is equal to the size of its convolutional kernel, so as to convert the ECG timing signal into a non-overlapping sequence of segments; the number of input channels of the one-dimensional convolution operation corresponds to the number of leads in the ECG. The location and category embedding layer is configured to perform: Concatenation operation: A learnable classification token vector is concatenated to the beginning position of the embedded vector sequence to form an extended sequence; Position encoding operation: Add an element-wise a learnable one-dimensional position encoding matrix to the extended sequence, wherein each row of the position encoding matrix corresponds to a position encoding vector at a specific position in the extended sequence; The multilayer Transformer encoder consists of multiple stacked residual attention blocks, each of which includes a layer normalization module, a multi-head self-attention mechanism module, residual connections, and a feedforward neural network.

4. The large-scale ECG diagnostic model architecture based on multimodal methods as described in claim 2, characterized in that, The ECG encoder also includes a projection head connected to the pooling layer, used to map the first feature representation to a preset shared embedding space dimension.

5. The large-scale ECG diagnostic model architecture based on multimodal methods as described in claim 1, characterized in that, The ECG timing signal is converted into the input ECG image using an ECG-image-kit; the image encoder is a pre-trained visual encoder based on the Vision Transformer architecture.

6. The large-scale ECG diagnostic model architecture based on multimodal methods as described in claim 1, characterized in that, The feature alignment module maps and merges the dimensions of the first feature representation and the second feature representation into a unified feature dimension; The feature alignment module is a multilayer perceptron, which is a two-layer perceptron containing two linear layers and a GELU activation function. During training, the parameters of the ECG encoder and the image encoder are frozen, and the parameters of the feature alignment module and the parameters of the large language model are jointly trained.

7. The large-scale ECG diagnostic model architecture based on multimodal methods as described in claim 1, characterized in that, The large language model receives fused feature representations from the feature alignment module and text instructions from external input, and generates structured text output related to electrocardiogram (ECG) diagnosis. The text instructions are text requests used to guide and limit the model to perform specific tasks. The text instructions from external input include at least one of the following types: instructions to request the generation of ECG diagnostic reports, instructions to ask questions to interpret ECG abnormalities, and instructions to request the interpretation of ECG features.

8. A training method for a large-scale ECG diagnostic model architecture based on multimodality, used to train the large-scale model architecture as described in any one of claims 1-7, characterized in that, It includes: Using a contrastive learning framework, the original ECG time-series signal and its corresponding medical text description are used as positive sample pairs for training. The ECG encoder is optimized so that its output first feature representation is aligned with the semantic representation of the medical text description in a shared embedding space. The original ECG timing signal is input into the pre-trained ECG encoder to obtain a first feature representation; the ECG image is input into the image encoder to obtain a second feature representation. The first feature representation and the second feature representation are fused and dimensionally aligned to generate a unified fused feature representation.

9. The training method for a large-scale ECG diagnostic model architecture based on multimodal methods as described in claim 8, characterized in that, The contrastive learning framework is the COCA framework, and its training objective is to simultaneously minimize the contrastive loss and the text description generation loss.

10. The training method for a large-scale ECG diagnostic model architecture based on multimodal methods as described in claim 9, characterized in that, The contrast loss includes signal-to-text contrast loss and text-to-signal contrast loss, which are calculated using a normalized exponential function and cross-entropy loss; the contrast loss is the mean of the sum of the signal-to-text contrast loss and the text-to-signal contrast loss; wherein: The signal-to-text comparison loss is used to maximize the similarity between ECG signal features and corresponding text features, while minimizing the similarity between the ECG signal features and non-corresponding text features. The text-to-signal contrast loss is used to maximize the similarity between text features and corresponding ECG signal features, while minimizing the similarity between the text features and non-corresponding ECG signal features.

Images