A method for learning electrocardiogram representations based on multimodal hybrid supervision

By employing a multimodal hybrid supervised ECG representation learning method, and utilizing the multimodal information from ECG and clinical reports, a text encoder and an ECG encoder are constructed. This achieves fine alignment and feature enhancement between ECG and clinical reports, solves the problem of poor reliability in existing technologies, and improves the accuracy of ECG classification.

CN123120342APending Publication Date: 2026-11-17TIANHE COLLEGE GUANGDONG POLYTECHNIC NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611169125.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-04
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing self-supervised learning methods for electrocardiograms are sensitive to negative sample selection, have weak overall semantic feature learning ability, and do not utilize the accompanying textual report information of electrocardiograms, resulting in poor reliability.

Method used

A multimodal hybrid supervision method is adopted, which utilizes the multimodal information of electrocardiogram and clinical report to construct a text encoder and an electrocardiogram encoder. They are jointly pre-trained through an independent gradient routing mechanism. By combining masked electrocardiogram modeling, masked language modeling, contrastive learning and binary cross-entropy loss, fine alignment and feature enhancement of electrocardiogram and clinical report are achieved.

Benefits of technology

It improves the reliability of ECG analysis, especially significantly enhancing the accuracy of ECG classification under low data conditions, and ensuring that the gain effects of different supervision sources do not cancel each other out.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123120342A_ABST
    Figure CN123120342A_ABST
Patent Text Reader

Abstract

This application relates to the technical field of intelligent analysis and provides a multimodal hybrid supervision-based electrocardiogram (ECG) representation learning method. The method includes first acquiring a first type of pre-training information and a second type of pre-training information, then constructing a text encoder and an ECG encoder, and finally activating the first and second training target set information based on this. This application enables joint optimization using two complementary supervision sources: free text clinical reports and structured diagnostic labels, while maintaining the independence of the two types of supervision gradients. This fully leverages the gain effects of each source on different diagnostic task types. Furthermore, it designs a signal block embedding mechanism for multi-lead structure perception to eliminate positional ambiguity in multi-lead modeling, establishes a fine correspondence between local ECG waveforms and clinical text descriptions at the signal block level, and prevents feature space collapse during joint training of heterogeneous data through a gradient routing isolation mechanism, effectively improving reliability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the technical field of intelligent analysis, and more specifically, to an electrocardiogram representation learning method based on multimodal hybrid supervision, which can be used in electrocardiogram automatic analysis and intelligent diagnosis systems, clinical cardiac auxiliary decision-making platforms, and wearable cardiac monitoring devices. Background Technology

[0002] Cardiovascular disease is the leading cause of death worldwide. A 12-lead electrocardiogram (ECG) is a non-invasive examination method that records the changes in cardiac electrical activity over time by placing multiple electrodes on the body surface. It can simultaneously acquire cardiac electrical signals in twelve different directions, with each direction corresponding to one lead. The 12-lead ECG is the most widely used cardiac diagnostic tool in clinical practice, used to assist in the diagnosis of various heart diseases such as arrhythmias, myocardial ischemia, and conduction blocks. However, accurate interpretation of ECGs relies heavily on the professional expertise of cardiologists. In areas with limited medical resources, the shortage of specialists severely restricts the early diagnosis and intervention of heart disease.

[0003] Deep learning technology provides an effective means for automating electrocardiogram (ECG) interpretation. Supervised deep learning models have achieved high performance in arrhythmia classification tasks, but they rely on large-scale manually labeled datasets, resulting in high labeling costs. Self-supervised learning (SSL) is a machine learning method that generates supervisory signals using the data's own structure without requiring manual labeling. It learns transferable feature representations using a large amount of unlabeled ECG data, making it a feasible way to overcome the labeling bottleneck.

[0004] Currently, self-supervised learning methods for electrocardiograms mainly include SimCLR, MoCo, BYOL, CLOCS, and ST-MEM. These methods are sensitive to the selection of negative samples and have relatively weak learning ability for overall semantic features. Furthermore, none of these methods utilize the accompanying textual report information of the electrocardiogram, resulting in poor reliability. Further improvements are needed. Summary of the Invention

[0005] Based on this, embodiments of this application provide an electrocardiogram representation learning method based on multimodal hybrid supervision to solve the problem of poor reliability in the prior art.

[0006] In a first aspect, embodiments of this application provide an electrocardiogram representation learning method based on multimodal hybrid supervision, the method comprising: Acquire a first type of pre-training information and a second type of pre-training information, wherein the first type of pre-training information is used to describe electrocardiogram recording information associated with clinical reports, and the second type of pre-training information is used to describe electrocardiogram recording information associated with diagnostic category labels; Construct a text encoder and an electrocardiogram encoder, wherein the text encoder is used to convert clinical reports into semantic feature representations, and the electrocardiogram encoder is used to convert the raw twelve-lead electrocardiogram signal into a high-dimensional feature representation; Based on the first type of pre-training information, the first training target set information is activated, and based on the second type of pre-training, the second training target set information is activated to perform joint pre-training by combining the text encoder and the electrocardiogram encoder. The gradients of the first training target set information and the second training target set information are independent of each other. The auxiliary classification loss in the second training target set information is routed to the feature path of the electrocardiogram encoder before cross-modal fusion to isolate it from the gradient of the contrast alignment path.

[0007] Compared with existing technologies, the beneficial effects are as follows: The ECG representation learning method based on multimodal hybrid supervision provided in this application allows the terminal device to first acquire first-type and second-type pre-training information, then construct a text encoder and an ECG encoder, and finally activate the first training target set information based on the first-type pre-training information and the second training target set information based on the second-type pre-training information. This achieves joint optimization using two complementary supervision sources: free text clinical reports and structured diagnostic labels, while maintaining the independence of the two types of supervision gradients. This fully explores the gain effects of each on different diagnostic task types. Furthermore, it designs a signal block embedding mechanism for multi-lead structure perception to eliminate positional ambiguity in multi-lead modeling, establishes a fine correspondence between ECG local waveforms and clinical text descriptions at the signal block level, and prevents feature space collapse during joint training of heterogeneous data through a gradient routing isolation mechanism. This effectively improves reliability and solves the problem of poor reliability in current methods to a certain extent.

[0008] Secondly, embodiments of this application provide an electrocardiogram representation learning system based on multimodal hybrid supervision, the system comprising: Pre-training information acquisition module: used to acquire a first type of pre-training information and a second type of pre-training information, wherein the first type of pre-training information is used to describe electrocardiogram recording information associated with clinical reports, and the second type of pre-training information is used to describe electrocardiogram recording information associated with diagnostic category labels; Encoder construction module: used to construct a text encoder and an electrocardiogram encoder, wherein the text encoder is used to convert clinical reports into semantic feature representations, and the electrocardiogram encoder is used to convert the raw twelve-lead electrocardiogram signal into a high-dimensional feature representation; Training target set information activation module: used to activate the first training target set information according to the first type of pre-training information, and activate the second training target set information according to the second type of pre-training information, so as to perform joint pre-training in combination with the text encoder and the electrocardiogram encoder, wherein the gradients of the first training target set information and the second training target set information are independent of each other, and the auxiliary classification loss in the second training target set information is routed to the feature path of the electrocardiogram encoder before cross-modal fusion, so as to isolate the gradient from the contrast alignment path.

[0009] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0010] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0011] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0013] Figure 1 This is a schematic flowchart of an electrocardiogram representation learning method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a multimodal electrocardiogram pre-training representation learning framework provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the lead sensing signal block embedding mechanism in an electrocardiogram representation learning method provided in an embodiment of this application; Figure 4 This is a flowchart illustrating step S300 in an electrocardiogram representation learning method provided in an embodiment of this application; Figure 5 This is a block diagram of an electrocardiogram representation learning system provided in one embodiment of this application; Figure 6 This is a schematic diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0014] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0015] In the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0016] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0017] To illustrate the technical solution described in this application, specific embodiments are provided below.

[0018] Please see Figure 1 , Figure 1 This is a flowchart illustrating the electrocardiogram (ECG) representation learning method based on multimodal hybrid supervision provided in this embodiment. In this embodiment, the execution entity of the ECG representation learning method is a terminal device. It is understood that the types of terminal devices include, but are not limited to, tablet computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc., and this embodiment does not impose any restrictions on the specific type of terminal device.

[0019] It should be noted that the existing technology has the following four main drawbacks: (1) The source of supervisory information is singular, and the clinical semantic information is not fully utilized. That is, the clinical semantics of electrocardiograms come from two different types of supervisory sources: free text reports (descriptive language containing waveform morphology) and structured diagnostic labels (concise encoding of disease and rhythm categories). Existing methods utilize at most one of them, and the supervisory signals contained in the other type are completely wasted.

[0020] (2) The multimodal alignment granularity is coarse and the local semantic correspondence is missing. That is, the existing multimodal electrocardiogram method only aligns the electrocardiogram and the text at the overall feature level, and cannot establish a fine semantic association between a specific lead and a specific time period of the electrocardiogram and the corresponding text description in the report, resulting in the learned feature representation lacking the ability to distinguish local morphology.

[0021] (3) Lack of signal block embedding design for multi-lead structure perception, i.e., the electrocardiogram is collected simultaneously by twelve leads. The signals of different leads at the same time have different physiological meanings. However, the existing method directly splices the signal blocks of each lead, failing to distinguish the signal identities of different leads at the same time, making it difficult for the model to accurately model the spatiotemporal coordination relationship across leads.

[0022] (4) When training heterogeneous data together, gradients interfere with each other, which can easily lead to feature space collapse. That is, when ECG-text pairs and ECG-structured label pairs are trained together, the gradient of the structured label classification loss will interfere with the contrast alignment path, causing the feature representations of different categories to lose their distinctiveness (feature space collapse). Existing methods have not provided an effective solution.

[0023] Please see Figure 1 The electrocardiogram representation learning method provided in this application includes, but is not limited to, the following steps: In S100, the first type of pre-training information and the second type of pre-training information are obtained.

[0024] Specifically, the terminal device can first obtain a first type of pre-training information and a second type of pre-training information. The first type of pre-training information is used to describe the electrocardiogram (ECG) recording information associated with the clinical report, and the second type of pre-training information is used to describe the ECG recording information associated with the diagnostic category label.

[0025] For example, please refer to Figure 2 The electrocardiogram (ECG) representation learning method of this application can serve as a multimodal ECG pre-training representation learning framework. During the pre-training phase, it simultaneously utilizes two types of heterogeneous data: Type A is ECG-text pair data, which includes ECG records accompanied by free-text clinical reports (Type A is the first type of pre-training information mentioned above); Type B is ECG-structured label pair data, which includes ECG records accompanied by diagnostic category labels (Type B is the second type of pre-training information mentioned above). The framework can consist of four core modules, corresponding to four training objectives, thereby jointly learning clinically meaningful ECG feature representations.

[0026] Without loss of generality, both the first and second training objective sets include Masked ECG Modeling (MEM) information. The first training objective set also includes Masked Language Modeling (MLM) and Contrastive Learning (CL) information, while the second training objective set also includes Binary Cross-Entropy (BCE) information.

[0027] It should be noted that the gradients of the first training objective set information and the second training objective set information are independent of each other. The auxiliary classification loss in the second training objective set information is routed to the feature path of the ECG encoder before cross-modal fusion to isolate it from the gradient of the contrast alignment path.

[0028] In S200, a text encoder and an electrocardiogram encoder are constructed.

[0029] Specifically, after the terminal device acquires the first type of pre-training information and the second type of pre-training information, the terminal device can effectively construct a text encoder and an electrocardiogram encoder. The text encoder is used to convert clinical reports into semantic feature representations, and the electrocardiogram encoder is used to convert the raw twelve-lead electrocardiogram signal into a high-dimensional feature representation.

[0030] For some possible implementations, please refer to Figure 3 The electrocardiogram encoder includes a lead sensing signal block embedding mechanism, which includes: In S210, the electrocardiogram recording signal of each lead is divided into multiple time segments along the time axis.

[0031] Specifically, the terminal device can first divide the ECG recording signal of each lead into multiple time segments along the time axis. The lead is used to describe any signal acquisition direction in the twelve-lead ECG. The time segments are of equal length, and each time segment is a signal block. All signal blocks of the twelve leads together constitute the input sequence of the ECG.

[0032] In S220, each signal block is assigned a content code, a lead identity code, and a time position code, and the content code, lead identity code, and time position code are superimposed to generate the final embedded representation information corresponding to each signal block.

[0033] Specifically, after the terminal device divides the information into multiple time segments, it can assign content encoding, lead identity encoding, and time position encoding to each signal block. Then, the content encoding, lead identity encoding, and time position encoding are superimposed to quickly generate the final embedded representation information corresponding to each signal block. Thus, the final embedded representation of each signal block is obtained after adding the three types of encoding. Among them, content encoding is used to map the waveform amplitude of the signal block into a fixed-dimensional feature vector through one-dimensional convolution; lead identity encoding is used to distinguish the signal identity of different leads at the same time, that is, assigning a learnable identity vector to each of the twelve leads and superimposing it on the features of all signal blocks of that lead, so that the model can clearly distinguish the signals of different leads at the same time; time position encoding is used to perceive the temporal order of the ECG recorded signals, that is, assigning a learnable position vector to the position of each signal block on the time axis and superimposing it on the features of that block, so that the model can perceive the temporal order of the signals.

[0034] It should be noted that the lead sensing signal block embedding mechanism embeds both lead identity and time location simultaneously, enabling any two signal blocks to be clearly distinguished by the model, regardless of whether they come from the same time in different leads. This eliminates positional ambiguity in multi-lead modeling and provides an unambiguous input basis for cross-lead dependency modeling.

[0035] In some possible implementations, after signal block embedding is completed, the terminal device can feed the entire signal block feature sequence into a multi-layer self-attention transform network for global context modeling. This multi-layer self-attention transform network, also known as a Transformer encoder, consists of multiple stacked self-attention layers and feedforward network layers. Each layer captures the dependencies between any two positions in the sequence through an attention mechanism. Simultaneously, the terminal device can add a learnable global convergence marker at the beginning of the sequence; its output represents the global feature representation of the entire electrocardiogram segment, used for subsequent comparison, alignment, and classification.

[0036] In some possible implementations, the text encoder can use a lightweight language model pre-trained on clinical medical text as a foundation, assign context-aware feature vectors to each word in the report, and extract global semantic representations at the report level. This allows for adaptation to the language style of ECG reports while retaining pre-trained language knowledge. A freezing strategy is adopted for the shallow parameters of the language model, allowing only deep parameters to participate in the update.

[0037] In some possible implementations, in order to introduce cross-modal attention fusion functionality, after step S200, the method may include, but is not limited to, the following steps: In S201, for the first type of pre-training information, the features of each signal block output by the ECG encoder are used as the query end, and the features of each word output by the text encoder are used as the key value end. Through cross-attention calculation, each signal block is made to pay attention to the semantically related word descriptions in the clinical report, so as to achieve signal block-level ECG and text semantic alignment.

[0038] Specifically, the cross-modal attention fusion function can inject semantic information from text reports into block-level features of electrocardiogram signals. See also... Figure 2 The terminal device can use the features of each signal block output by the ECG encoder as the query end and the features of each word output by the text encoder as the key end for the first type of pre-training information. Through cross-attention calculation, each signal block can selectively focus on semantically related word descriptions in the clinical report to achieve signal block-level ECG and text semantic alignment.

[0039] In some possible implementations, to prevent the ECG encoder from becoming dependent on text reports during pre-training and affecting the performance of downstream single-modal inference, a modality random discarding mechanism can be introduced. After step S200, the method also includes, but is not limited to, the following steps: In S202, during training, the first type of pre-training information is randomly skipped with a preset probability. The features of each signal block output by the ECG encoder are used as the query end, and the features of each word output by the text encoder are used as the key end. Through cross-attention calculation, each signal block is made to focus on the semantically related word descriptions in the clinical report. Then, the output of the ECG encoder is used directly to ensure that the ECG encoder maintains its single-modal feature extraction capability without text assistance.

[0040] Specifically, during training, the terminal device can randomly skip step S201 with a preset probability and then directly use the output of the ECG encoder to ensure that the ECG encoder maintains its unimodal feature extraction capability without text assistance. In S300, the first training target set information is activated based on the first type of pre-training information, and the second training target set information is activated based on the second type of pre-training information, so as to perform joint pre-training by combining the text encoder and the electrocardiogram encoder.

[0041] Specifically, after the terminal device constructs the text encoder and the electrocardiogram encoder, the terminal device can effectively activate the first training target set information based on the first type of pre-training information, and effectively activate the second training target set information based on the second type of pre-training information, so as to combine the text encoder and the electrocardiogram encoder for joint pre-training, thereby completing the joint optimization of four types of training targets (i.e., MEM, MLM, CL and BCE). In some possible implementations, to achieve joint optimization of the four training objectives, please refer to [link / reference]. Figure 4 Step S300 includes, but is not limited to, the following steps: In S310, the signals of any one or more complete leads are masked as a whole, and the signal waveforms of the masked leads are reconstructed by the decoder based on the signal block characteristics of the visible leads, so as to activate the masked ECG modeling information.

[0042] Specifically, the terminal device can mask the signal of any one or more complete leads as a whole, and based on the signal block characteristics of the visible leads, reconstruct the signal waveform of the masked leads through the decoder to activate the masked ECG modeling information and complete the ECG mask reconstruction. The reconstruction error during signal waveform reconstruction is composed of a weighted sum of three components: time domain waveform error, frequency domain amplitude spectrum error, and signal gradient error.

[0043] Without loss of generality, time-domain waveform error is used to measure the point-by-point difference between the reconstructed waveform and the original waveform, frequency-domain amplitude spectrum error is used to measure the difference between the reconstructed spectrum and the original spectrum and to monitor the frequency characteristics of P-waves, QRS groups, and T-waves, and signal gradient error is used to measure the difference between the reconstructed waveform and the original waveform in terms of first-order slope and second-order curvature.

[0044] For example, the terminal device can mask the signals of several complete leads of the input electrocardiogram by lead as a whole. Based on the signal block characteristics of the visible leads, the signal waveform of the masked leads is reconstructed by a lightweight decoder. This objective forces the encoder to capture the local time-frequency structure and translead compensation rules of the signal. The reconstruction error is comprehensively measured by three components: (1) time domain waveform error, which measures the point-by-point difference between the reconstructed waveform and the original waveform; (2) frequency domain amplitude spectrum error, which measures the difference between the reconstructed spectrum and the original spectrum and supervises different frequency characteristics such as P wave, QRS complex, and T wave; and (3) signal gradient error, which measures the difference between the reconstructed waveform and the original waveform in the first-order slope and second-order curvature, and is particularly sensitive to the sharpness of the QRS complex and the curvature of the ST segment. The complete reconstruction loss is constituted by weighted summation of the three components.

[0045] In S320, for the first type of pre-trained information, any one or more words are randomly masked, and the text encoder is trained to predict the original content of the masked words through context in order to activate the masked language modeling information.

[0046] Specifically, the terminal device can randomly mask any one or more words based on the first type of pre-training information, train the text encoder to predict the original content of the masked words through context, activate the masked language modeling information, and complete the text masked language modeling.

[0047] For example, the terminal device can randomly mask some words in clinical reports from Class A data, train the text encoder to predict the original content of the masked words through context, and make the text encoder adapt to the morphological descriptive language style of the electrocardiogram report.

[0048] In S330, for the first type of pre-training information, based on the projection representation of the global convergence labels of the electrocardiogram and the text, the feature vectors of positive sample pairs of the same record are brought closer to each other, and the feature vectors of negative sample pairs of different records are moved further apart. A preset bidirectional calculation method is used to activate the contrastive learning information.

[0049] Specifically, the terminal device can, for the first type of pre-training information, use the projected representation of the global convergence labels of the electrocardiogram and the text as a basis to bring the feature vectors of positive sample pairs of the same record closer together and make the feature vectors of negative sample pairs of different records farther apart. It can also use a preset bidirectional calculation method to activate the contrast learning information and complete the bidirectional comparison and alignment of the electrocardiogram and the text. The bidirectional calculation method optimizes both the retrieval of text from the electrocardiogram and the retrieval of electrocardiogram from the text.

[0050] In S340, for the second type of pre-training information, based on the global convergence label output by the ECG encoder before cross-modal fusion, the presence or absence of each structured diagnostic category is predicted to activate the binary cross-entropy information.

[0051] Specifically, the terminal device can use the global convergence label output by the ECG encoder before cross-modal fusion to predict the presence or absence of each structured diagnostic category through a multi-layer sensing mechanism for the second type of pre-training information. This activates the binary cross-entropy information and completes auxiliary multi-label classification, thereby providing the encoder with a clear rhythm-based diagnostic discrimination supervision signal. The structured diagnostic categories can cover both rhythm and disease categories.

[0052] It should be noted that the loss in step S240 is routed to the feature path before fusion to achieve gradient isolation with the contrast-aligned path.

[0053] In some possible implementations, to further improve reliability, after step S300, the method may include, but is not limited to, the following steps: In S400, the reconstruction target value for each signal block is normalized within the block to prevent high-amplitude QRS wave dominance loss from ignoring the reconstruction quality of P-waves and T-waves.

[0054] Specifically, the terminal device can normalize the reconstruction target value of each signal block within the block to prevent high-amplitude QRS wave dominance loss from ignoring the reconstruction quality of clinically relevant morphologies such as P wave and T wave.

[0055] In some possible implementations, in order to complete the joint calculation of route-aware heterogeneous losses, after step S300, the method may include, but is not limited to, the following steps: In S500, the loss value of each training objective is multiplied by its own fixed weight, summed, and then divided by the sum of all fixed weights to obtain the total training loss. A fixed denominator ensures that the total loss scale is stable under heterogeneous batches.

[0056] Specifically, the terminal device can multiply the loss value of each training objective by its own fixed weight, sum them up, and then divide by the sum of all fixed weights to obtain the total training loss. The fixed denominator ensures that the total loss scale is stable under heterogeneous batches, where the gradient contributions of each supervision source are independent of each other.

[0057] In some possible implementations, to adapt to downstream tasks, the text encoder, fusion module, and decoder can all be discarded after pre-training, retaining only the ECG encoder for downstream tasks such as ECG classification and diagnosis. It should be noted that this modality random discarding mechanism ensures that the encoder maintains stable feature extraction capabilities in pure signal inference scenarios without text.

[0058] It should be noted that the main reasons for the performance advantages of this application include, but are not limited to: (1) The introduction of structured label supervision provides discriminative signals that are difficult to be fully covered by text reports for rhythmic tasks, and its compensatory effect is particularly significant when labeled data is scarce; (2) Fine alignment of signal block granularity enables the encoder to establish a direct association between clinical text descriptions (e.g., ST segment depression, T wave inversion) and the corresponding local waveforms, thereby improving the discriminative ability of fine-grained morphological tasks; (3) The gradient isolation mechanism can prevent structured label supervision from interfering with the feature alignment space and ensure that the gain effects of different supervision sources do not cancel each other out.

[0059] In some possible implementations, the pre-training data used in this application can be the MIMIC-IV-ECG dataset and the HEEDB dataset, and then the two types of data are sampled in an approximately 1:1 ratio. The MIMIC-IV-ECG dataset can contain approximately 298,000 ECG-text pairs, all of which are 10-second twelve-lead datasets with a sampling rate of 500Hz. The HEEDB dataset can contain 503 types of structured multi-thermal diagnostic labels.

[0060] An example parameter configuration is as follows: The ECG encoder uses a 6-layer Transformer network with a hidden layer dimension of 384 and includes 6 attention heads. Each lead is divided into a signal block with 50 sampling points, and a total of 1200 signal blocks are generated for 12 leads; the cross-modal fusion module can include 2 layers of cross-attention blocks; the hidden layer dimension of the text encoder is projected to 384 dimensions.

[0061] The loss weights for each training objective can be as follows: masked ECG modeling loss is 1.0, masked language modeling loss is 0.4, contrastive learning loss is 0.5, and auxiliary classification loss is 0.3; the modality random dropout rate is set to 30%.

[0062] It should be noted that this method was systematically evaluated on six publicly available ECG classification benchmark datasets (i.e., PTB-XL superclass, subclass, morphology class, rhythm class, CPSC2018, and CSN), using a linear evaluation protocol (i.e., freezing encoder parameters and training only the linear classification head), and the following main conclusions were drawn: (1) Compared with the baseline of pre-training reconstructed using only masks, the introduction of two types of supervision sources improved the average AUC (area under the curve) by about 17 percentage points in the low data scenario with 1% labeled data and by about 12 percentage points in the 100% data scenario. Multimodal supervision is most valuable when labels are scarce. (2) After removing text report supervision, the performance of morphological classification tasks consistently decreased; after removing structured label supervision, the performance of rhythmic tasks decreased, especially when the amount of data was low; the gain effects of the two sources did not overlap, verifying the necessity of using them together.

[0063] (3) It outperforms the known best solution in most configurations of the six datasets. In the low data volume (i.e. 1% labeled) scenario, it leads by an average AUC of about 4 percentage points and in the full data scenario, it leads by about 3 percentage points.

[0064] In some possible implementations, this application employs a lightweight clinical domain language model for the text encoder. This model can also be replaced with other language models pre-trained on biomedical or clinical text (such as Bio_ClinicalBERT or PubMedBERT), or with an encoder-decoder architecture language model. The selection principle is to effectively encode the clinical morphological descriptive language in electrocardiogram reports. In some possible implementations, this application adopts bidirectional InfoNCE contrastive loss for the contrastive alignment loss function. It can also be replaced by other contrastive learning objective functions such as sigmoid-based full-pair loss and NT-Xent loss. The core is to establish a discriminative alignment relationship between the electrocardiogram and the global text representation.

[0065] In some possible implementations, this application adopts a lead-based overall masking strategy for signal block masking, which can also be replaced by a time-based randomized masking strategy, or a combination of the two. Lead-based overall masking forces the model to make full use of cross-lead compensation information, while randomized masking has a lower computational cost.

[0066] In some possible implementations, for cross-modal fusion, this application implements cross-attention fusion at the signal block level, and can also implement lightweight gated fusion or feature stitching projection at the global convergence label level. Signal block-level fusion has higher accuracy in local semantic alignment, while global-level fusion has lower computational cost.

[0067] In some possible implementations, for the source of structured labeled data, this application uses a dataset containing 503 diagnostic labels to provide auxiliary classification supervision. Alternatively, it can be replaced with other publicly available electrocardiogram datasets with multi-label diagnostic annotations. The core is to provide structured discrimination supervision for rhythm and disease categories.

[0068] The implementation principle of the ECG representation learning method based on multimodal hybrid supervision in this application is as follows: The terminal device can first acquire the first type of pre-training information and the second type of pre-training information, then construct a text encoder and an ECG encoder, and finally activate the first training target set information according to the first type of pre-training information and activate the second training target set information according to the second type of pre-training information. This achieves joint optimization using two complementary supervision sources: free text clinical reports and structured diagnostic labels, while maintaining the independence of the two types of supervision gradients. This fully explores the gain effects of each on different diagnostic task types, and designs a signal block embedding mechanism for multi-lead structure perception to eliminate positional ambiguity in multi-lead modeling. Furthermore, it establishes a fine correspondence between ECG local waveforms and clinical text descriptions at the signal block level, and prevents feature space collapse during joint training of heterogeneous data through a gradient routing isolation mechanism, effectively improving reliability.

[0069] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0070] Embodiments of this application also provide an electrocardiogram representation learning system based on multimodal hybrid supervision. For ease of explanation, only the parts relevant to this application are shown, such as... Figure 5 As shown, the system 50 includes: Pre-training information acquisition module 51: used to acquire first type of pre-training information and second type of pre-training information, wherein the first type of pre-training information is used to describe electrocardiogram recording information associated with clinical reports, and the second type of pre-training information is used to describe electrocardiogram recording information associated with diagnostic category labels; Encoder building module 52: used to build a text encoder and an electrocardiogram encoder, wherein the text encoder is used to convert clinical reports into semantic feature representations, and the electrocardiogram encoder is used to convert the raw twelve-lead electrocardiogram signal into a high-dimensional feature representation; Training target set information activation module 53: is used to activate the first training target set information according to the first type of pre-training information, and to activate the second training target set information according to the second type of pre-training information, so as to combine the text encoder and the electrocardiogram encoder for joint pre-training. The gradients of the first training target set information and the second training target set information are independent of each other. The auxiliary classification loss in the second training target set information is routed to the feature path of the electrocardiogram encoder before cross-modal fusion, so as to isolate the gradient from the contrast alignment path.

[0071] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0072] This application also provides a terminal device, such as... Figure 6 As shown, the terminal device 60 in this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the processor 61 executes the computer program 63, it implements the steps described in the above-described electrocardiogram representation learning method embodiment, for example... Figure 1 Steps S100 to S300 are shown; or, when processor 61 executes computer program 63, it implements the functions of each module in the above-described device, for example... Figure 5 The functions of modules 51 to 53 are shown.

[0073] The terminal device 60 can be a desktop computer, laptop, handheld computer, or cloud server, etc., and includes, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 6 This is merely an example of terminal device 60 and does not constitute a limitation on terminal device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, terminal device 60 may also include input / output devices, network access devices, buses, etc.

[0074] The processor 61 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.; the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0075] The memory 62 can be an internal storage unit of the terminal device 60, such as the hard disk or memory of the terminal device 60. The memory 62 can also be an external storage device of the terminal device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 60. Furthermore, the memory 62 can include both internal storage units and external storage devices of the terminal device 60. The memory 62 can also store computer program 63 and other programs and data required by the terminal device 60. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0076] One embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0077] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the methods, principles and structures of this application should be covered within the scope of protection of this application.

Claims

1. A method for learning electrocardiogram representations based on multimodal hybrid supervision, characterized in that, The method includes: Acquire a first type of pre-training information and a second type of pre-training information, wherein the first type of pre-training information is used to describe electrocardiogram recording information associated with clinical reports, and the second type of pre-training information is used to describe electrocardiogram recording information associated with diagnostic category labels; A text encoder and an electrocardiogram encoder are constructed, wherein the text encoder is used to convert clinical reports into semantic feature representations, and the electrocardiogram encoder is used to convert the raw twelve-lead electrocardiogram signal into a high-dimensional feature representation; Based on the first type of pre-training information, the first training target set information is activated, and based on the second type of pre-training, the second training target set information is activated to perform joint pre-training by combining the text encoder and the electrocardiogram encoder. The gradients of the first training target set information and the second training target set information are independent of each other. The auxiliary classification loss in the second training target set information is routed to the feature path of the electrocardiogram encoder before cross-modal fusion to isolate it from the gradient of the contrast alignment path.

2. The method according to claim 1, characterized in that, Both the first training objective set information and the second training objective set information include masked electrocardiogram modeling information. The first training objective set information also includes masked language modeling information and contrastive learning information. The second training objective set information also includes binary cross-entropy information.

3. The method according to claim 1, characterized in that, The electrocardiogram encoder includes a lead sensing signal block embedding mechanism, wherein the lead sensing signal block embedding mechanism includes: The electrocardiogram recording signal of each lead is divided into multiple time segments along the time axis. The lead is used to describe any signal acquisition direction in the twelve-lead electrocardiogram. The time segments are of equal length and each time segment is a signal block. Each signal block is assigned a content code, a lead identity code, and a time position code. The content code, lead identity code, and time position code are then superimposed to generate the final embedded representation information corresponding to each signal block. The content code is used to map the waveform amplitude of the signal block to a feature vector of fixed dimension. The lead identity code is used to distinguish the signal identity of different leads at the same time. The time position code is used to perceive the temporal order of the electrocardiogram recording signal.

4. The method according to claim 1, characterized in that, After constructing the text encoder and the electrocardiogram encoder, the method further includes: For the first type of pre-training information, the features of each signal block output by the ECG encoder are used as the query end, and the features of each word output by the text encoder are used as the key value end. Through cross-attention calculation, each signal block is made to pay attention to the semantically related word descriptions in the clinical report, so as to achieve signal block-level ECG and text semantic alignment.

5. The method according to claim 4, characterized in that, After constructing the text encoder and the electrocardiogram encoder, the method further includes: During training, the pre-training information for the first type is randomly skipped with a preset probability. The features of each signal block output by the ECG encoder are used as the query end, and the features of each word output by the text encoder are used as the key value end. Through cross-attention calculation, each signal block is made to focus on the semantically related word descriptions in the clinical report. Then, the output of the ECG encoder is used directly to ensure that the ECG encoder maintains its single-modal feature extraction capability without text assistance.

6. The method according to claim 1, characterized in that, The step of activating the first training target set information according to the first type of pre-training information and activating the second training target set information according to the second type of pre-training information, in order to perform joint pre-training of the text encoder and the electrocardiogram encoder, includes: The signal of any one or more complete leads is masked as a whole, and the signal waveform of the masked lead is reconstructed by a decoder based on the signal block characteristics of the visible leads to activate the masked ECG modeling information. The reconstruction error during signal waveform reconstruction is composed of a weighted sum of three components: time domain waveform error, frequency domain amplitude spectrum error, and signal gradient error. The time domain waveform error is used to measure the point-by-point difference between the reconstructed waveform and the original waveform. The frequency domain amplitude spectrum error is used to measure the difference between the reconstructed spectrum and the original spectrum and to supervise the frequency characteristics of the P wave, QRS complex, and T wave. The signal gradient error is used to measure the difference between the reconstructed waveform and the original waveform in terms of first-order slope and second-order curvature. For the first type of pre-trained information, randomly mask one or more words, and train the text encoder to predict the original content of the masked words through context, so as to activate the masked language modeling information; For the first type of pre-training information, based on the projection representation of the global convergence markers of electrocardiogram and text respectively, the feature vectors of positive sample pairs of the same record are brought closer to each other, and the feature vectors of negative sample pairs of different records are moved further apart. A preset bidirectional calculation method is used to activate the contrastive learning information. For the second type of pre-training information, based on the global convergence label output by the ECG encoder before cross-modal fusion, the presence or absence of each structured diagnostic category is predicted to activate the binary cross-entropy information.

7. The method according to claim 1, characterized in that, After activating the first training target set information according to the first type of pre-training information and activating the second training target set information according to the second type of pre-training information to perform joint pre-training of the text encoder and the electrocardiogram encoder, the method further includes: The reconstruction target value for each signal block is normalized within the block to prevent high-amplitude QRS wave dominance loss from ignoring the reconstruction quality of P and T waves.

8. The method according to claim 1, characterized in that, After activating the first training target set information according to the first type of pre-training information and activating the second training target set information according to the second type of pre-training information to perform joint pre-training of the text encoder and the electrocardiogram encoder, the method further includes: The total training loss is obtained by multiplying the loss value of each training objective by its own fixed weight, summing the results, and then dividing by the sum of all fixed weights. A fixed denominator ensures that the total loss scale is stable under heterogeneous batches.

9. A multimodal hybrid supervision-based electrocardiogram representation learning system, characterized in that, The system includes: Pre-training information acquisition module: used to acquire a first type of pre-training information and a second type of pre-training information, wherein the first type of pre-training information is used to describe electrocardiogram recording information associated with clinical reports, and the second type of pre-training information is used to describe electrocardiogram recording information associated with diagnostic category labels; Encoder construction module: used to construct a text encoder and an electrocardiogram encoder, wherein the text encoder is used to convert clinical reports into semantic feature representations, and the electrocardiogram encoder is used to convert the raw twelve-lead electrocardiogram signal into a high-dimensional feature representation; Training target set information activation module: used to activate the first training target set information according to the first type of pre-training information, and activate the second training target set information according to the second type of pre-training information, so as to perform joint pre-training in combination with the text encoder and the electrocardiogram encoder, wherein the gradients of the first training target set information and the second training target set information are independent of each other, and the auxiliary classification loss in the second training target set information is routed to the feature path of the electrocardiogram encoder before cross-modal fusion, so as to isolate the gradient from the contrast alignment path.