A protocol guide-based modal adaptive electrocardio world model construction method

Abstract

The application discloses a protocol-guided and modality-adaptive electrocardiogram world model construction method, and belongs to the technical field of electrocardiogram analysis and processing. Heartbeats are cut from original signals and spliced into electrocardiogram sentences, an electrocardiogram vocabulary table is constructed through vector quantization reconstruction, and a pre-training time sequence encoder is obtained through self-supervised pre-training of mask sentence prediction. A multi-stage electrocardiogram interpretation protocol is constructed, and an interpretation corpus is generated through protocol guidance. A decoupled bimodal architecture with the pre-training encoder as the time sequence encoder is adopted, and staggered modal dropout is applied in training to minimize the cross-environment mixed risk, so that a protocol-guided and modality-adaptive electrocardiogram world model is constructed. The electrocardiogram world model can output a structured interpretation including an inference process and a final diagnosis for the input electrocardiogram. The application significantly reduces the interpretation illusion, and enables the model to maintain high diagnostic performance and cross-modal consistency when any modality of the electrocardiogram signal or image is missing.

Description

Technical Field

[0001] This invention belongs to the field of electrocardiogram (ECG) analysis and processing technology, and involves the intersection of artificial intelligence and medical information processing technologies. Specifically, it relates to a protocol-guided modal adaptive ECG world model construction method. Background Technology

[0002] An electrocardiogram (ECG) collects the electrical activity of the heart using electrodes on the body surface. It is a fundamental diagnostic tool for arrhythmias and other heart diseases, and also the most common form of data in clinical practice. The reliability of ECG interpretation is crucial; incorrect interpretation can directly affect subsequent diagnosis and treatment. Clinically, ECGs are presented in two forms: time-series signals and images, both expressing the same waveform and carrying consistent meaning.

[0003] Existing self-supervised representation learning for electrocardiograms (ECGs) has limitations. Supervised deep learning methods rely on large-scale, expert-reviewed, high-quality labeled data and are typically designed for specific tasks, limiting their generalization ability. Therefore, ECG self-supervised learning learns general representations through pre-training on a large number of unlabeled records and then fine-tuning them for downstream tasks. Existing self-supervised methods can be divided into contrastive and reconstruction methods, but almost all treat ECGs as ordinary time series, using fixed-size, fixed-step time windows to segment the signal. This leads to two drawbacks: First, it ignores the morphological and rhythmic features of ECGs—for example, myocardial infarction needs to be judged based on ST segment elevation of a single heartbeat, and atrial fibrillation needs to be judged based on the overall rhythm. Fixed-window segmentation cannot characterize the unique morphology and rhythm of ECGs, reducing the effectiveness of representation learning. Second, it ignores the potential semantic relationships in ECGs—significant differences in heart rate exist between different individuals and even between different samples of the same individual. Fixed-window segmentation causes significant misalignment between samples, destroying the potential semantic relationships between heartbeats. In addition, existing ECG language processing methods mostly segment different waveforms within the heartbeat to construct a vocabulary. When the signal quality is poor, it is difficult to accurately segment fine-grained waveforms, and the vocabulary size is very small, usually not exceeding seventy categories, which limits the richness of semantic information.

[0004] In recent years, most general and medical multimodal large language models only support image-based electrocardiogram (ECG) analysis, not time-series signal analysis. Furthermore, they exhibit significant hallucinations in routine clinical ECGs—their interpretations appear structurally complete and use professional terminology, but quantitative assessments reveal numerous factual inaccuracies and low diagnostic accuracy. The fundamental limitation of intelligent ECG interpretation lies in the scarcity of high-quality interpretation corpora: existing corpora are mostly generated from labels and features by pure cue-based general large language models, driven by pre-trained priors rather than explicit diagnostic criteria. This easily introduces clinically invalid causal attributions and factual errors, and suffers from defects such as violating basic ECG principles, incomplete diagnostic dimensions, and superficial imitation of labeled data while ignoring unlabeled abnormalities. On the other hand, existing full-perception models fusing signals and images employ coupled coding: tightly coupling temporal coding to image placeholders and implicitly assuming paired modalities, and reusing a single projection layer across heterogeneous modalities. This leads to a significant decrease in diagnostic performance when a particular modality is missing, and inconsistencies between the outputs of the two single modalities. In summary, existing technologies have significant shortcomings in both the representation learning and intelligent interpretation of electrocardiograms. There is a lack of a unified solution that can learn morphological and rhythmic general representations from unlabeled data and achieve low-illusion, modal adaptive, and reliable interpretation based on these representations. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a protocol-guided modal adaptive ECG world model construction method. First, morphological and rhythmic representations of cardiac electrical activity are learned from unlabeled ECGs in a self-supervised manner to obtain a pre-trained temporal encoder (i.e., the base of the ECG world model). Then, based on the pre-trained temporal encoder, a protocol-guided, modal adaptive cardiac electrical activity world model (ECG world model) is constructed by combining protocol-guided data generation and modal adaptive training. The constructed ECG world model can be used to output a low-illusion, evidence-grounded structured interpretation of the input ECG.

[0006] For convenience, the relevant terms in this invention are defined as follows:

[0007] A world model is a model that learns the internal representation of the environment in a self-supervised manner and can be used for prediction or generation. Existing work includes World Models proposed by Ha and Schmidhuber, as well as the BERT / BEiT paradigm that learns the internal representation by predicting with a mask.

[0008] The cardiac electrical activity world model (hereinafter referred to as the ECG world model) mentioned in this invention refers to a model that internally represents and predicts cardiac electrical activity through self-supervised learning. This invention is the first to apply the world model to cardiac electrical activity: using heartbeats as "words", a discrete set vocabulary to characterize basic morphological states, and mask prediction to characterize rhythm dynamics, thereby constructing the ECG world model.

[0009] The technical solution provided by this invention is as follows:

[0010] A protocol-guided modal adaptive electrocardiogram world model construction method, comprising:

[0011] 1) Steps for building a pre-trained temporal encoder:

[0012] The system detects QRS waves (QRS complex, a collective term for Q waves, R waves, and S waves representing ventricular depolarization in an electrocardiogram) from the raw electrocardiogram signal using a heartbeat segmenter and segments the heartbeat into electrocardiogram words. These words from multiple leads are then sequentially concatenated to form an electrocardiogram sentence. Through vector quantization and reconstruction of the heartbeats, an electrocardiogram word list is constructed, consisting of several discrete sets of electrocardiogram words—a discrete heartbeat representation unit. Specifically, the heartbeats are mapped to ECG word vector representations through vector quantization and reconstruction. A portion of the electrocardiogram words in the sentence are masked, and the corresponding set of ECG words for the masked portion is predicted based on the unmasked context. The morphological and rhythmic representations of the electrocardiogram signal are learned in a self-supervised manner, resulting in a pre-trained temporal encoder. This self-supervised pre-training process yields the model backbone.

[0013] 2) Protocol bootstrapping data generation steps:

[0014] This invention incorporates clinical diagnostic thinking and constructs a multi-stage electrocardiogram (ECG) interpretation protocol; during the reasoning process, the ECG model strictly executes this protocol to interpret the ECG.

[0015] The ECG feature extractor extracts structured physiological features from the raw ECG signal, one lead at a time and one heartbeat at a time, to obtain a feature dictionary. Based on a pre-constructed multi-stage ECG interpretation protocol containing quantitative thresholds and systematic diagnostic logic, the feature dictionary is used to assemble protocol perception prompts. The protocol perception prompts are then input into the interpretation generation model (an arbitrary general-purpose large language model) to generate ECG interpretations that follow a fixed template. The fixed template includes a thought chain part that encodes the multi-step protocol reasoning trajectory, a summary part, and an answer part that gives the final diagnosis. The above process is repeated to obtain interpretation corpus, which is used as training corpus.

[0016] 3) Model building steps: Construct an electrocardiogram (ECG) world model from the perspective of using heartbeats as words and rhythms as sentences.

[0017] This invention discretizes heartbeats into a finite set of ECG words forming a vocabulary, equivalent to the basic morphological state space of this "world." By predicting the set of ECG words corresponding to masked heartbeats from context, it learns the rhythmic dynamics between heartbeats in an ECG sentence, equivalent to the state transition rules of this world. Thus, it simultaneously characterizes the morphological (individual heartbeat morphology) and rhythmic (inter-heartbeat sequence semantics) structures. This ECG world model can reconstruct and predict cardiac electrical activity patterns, providing a robust and semantically grounded representation for downstream interpretation.

[0018] An ECG world model is constructed, comprising an independent temporal encoder and an image encoder (for extracting representations from ECG images and handling modal heterogeneity independently of the temporal encoder) and independent projection layers (for connecting the multimodal encoder (including the temporal encoder and the image encoder) to a linear transformation layer of a unified semantic space, maintaining the integrity of multimodal-specific information), wherein the temporal encoder is the pre-trained temporal encoder; the model sets explicit ECG labels and injects temporal tokens obtained by the temporal encoder only at the ECG labels; the ECG world model is used for the structured interpretation of the input ECG output, including multi-step inference trajectories and the final diagnosis.

[0019] 4) Modality adaptive training steps:

[0020] Using the aforementioned interpretation corpus, during the training phase, a sampling transformation is applied to the model input from a finite set of environments according to a sampling distribution. This finite set of environments includes four types: discarding image modalities, discarding temporal modalities, maintaining the canonical order of modal token blocks, and swapping the order of modal token blocks. The sampling distribution is induced by two independent trials: a modal discarding trial with a modal discarding probability (first probability) and a token order swapping trial with a token order swapping probability (second probability) while retaining both modalities. The model parameters are optimized with the objective of minimizing the cross-environment mixing risk under the sampling distribution, resulting in a modality-adaptive ECG world model. The first probability ranges from the open interval (0,1), preferably 0.5. The second probability also ranges from the open interval (0,1), preferably 0.1.

[0021] The environmental sampling probability is induced by the first and second probabilities. The coverage condition theory guarantees that the model is fully trained under various modal input conditions during modal adaptive training, thus ensuring that the final model can still work reliably in actual clinical scenarios where any modality is missing.

[0022] After a modal adaptive training step, interleaved modal dropout training is applied to the ECG model to finally obtain a modal adaptive ECG world model. Interleaved modal dropout specifically involves: explicitly defining a finite set of environments containing four types of environments: "discarding images, discarding time sequences, canonical order, and exchanging order"; inducing a sampling distribution through two independent trials: "performing a modal dropout trial with a first probability" and "performing an order exchange trial with a second probability"; and minimizing the risk of cross-environment mixing as the training objective, with theoretical guarantees of robustness and consistency.

[0023] For the protocol-guided modal adaptive ECG world model construction method, further, the heartbeat is segmented into ECG words, including: after bandpass filtering the signal of a lead, the signal is integrated by moving wave integration to obtain an integrated signal; the local maxima of the integrated signal are traversed, and the local maxima that are after the refractory period and exceed the detection threshold are determined as QRS waves; for each lead, the ECG words corresponding to the heartbeat are segmented by using the midpoint of the adjacent QRS wave index as the interval boundary, and ECG words that are less than the set time window length are padded with zeros; the ECG words of multiple leads are sequentially concatenated to form an ECG sentence, and the length of the ECG sentence is unified to the set length, and those that are insufficient are padded with zeros and those that are excessive are truncated.

[0024] For the protocol-guided modal adaptive ECG world model construction method, the construction of the ECG vocabulary further includes: obtaining several representations from ECG sentences through the temporal encoder; quantizing each representation by finding the nearest neighbor set of ECG words in the ECG vocabulary based on cosine similarity (the codebook learning mechanism of VQ-VAE can be used); reconstructing the original heartbeat signal from the quantized set of ECG words through the decoder; jointly optimizing the quantization process and the reconstruction process using the reconstruction error and the loss consisting of the distance between the representation and the set of ECG words; and updating the ECG vocabulary using an exponential moving average strategy.

[0025] For the protocol-guided modal adaptive ECG world model construction method, further, the prediction of the set of ECG words corresponding to the masked part based on the unmasked context includes: randomly masking the ECG words in the ECG sentence according to a set masking ratio, replacing the masked ECG words with learnable masking labels, inputting the ECG representation obtained by the pre-trained temporal encoder into a linear classifier (such as using a BERT / BEiT-type masking prediction head) to predict the index of the set of ECG words corresponding to the masked ECG words, and using the difference between the predicted index and the true index as the training target; the temporal encoder maps each ECG word to a word through one-dimensional convolution, and then superimposes spatial embedding, temporal embedding and positional embedding before inputting it into the transformer encoder, wherein the spatial embedding maps each ECG word back to its lead, and the temporal embedding maps each ECG word to the time interval of its QRS wave.

[0026] For the protocol-guided modal adaptive ECG world model construction method, further, the structured physiological features include heart rate, RR interval, P wave amplitude and duration, PR interval, QRS amplitude and duration, T wave amplitude and duration, ST segment descriptor, and QT / QTc interval. For each of the twelve leads, multiple feature sequences are constructed in time order. The multi-stage ECG interpretation protocol is reorganized from the ECG interpretation process in clinical medical monographs into five stages, namely, technical quality control-heart rate-rhythm stage, conduction-electric axis-interval stage, chamber hypertrophy-voltage stage, ischemia-infarction-differentiation stage, and electrolyte-QT stage.

[0027] For the protocol-guided modal adaptive ECG world model construction method, further, the protocol-aware prompts include differential exclusion instructions to exclude differential diagnoses with explicit negative findings. The quantitative thresholds include at least one of the following: left ventricular hypertrophy voltage standard, normal PR interval range, QRS duration threshold, ST segment elevation threshold, pathological Q wave standard, and corrected QT interval upper limit. The multi-step protocol inference trajectory consists of six steps, including a five-step analysis corresponding to the five stages and a sixth step of final medical inference. The process of generating ECG interpretation adopts a blind interpretation simulation method, that is, interpretation is performed without explicitly mentioning the existence of the reference report, and the inference of the interpretation is aligned with the internal ECG report used as the basis for the answer, thereby identifying unlabeled abnormalities while conforming to the ECG report.

[0028] For the protocol-guided modal adaptive ECG world model construction method, further, the sampling probabilities of the four environments satisfy the following: the probability of discarding the image modality and the probability of discarding the temporal modality are both half of the first probability, the probability of maintaining the normal order is (1 minus the first probability) multiplied by (1 minus the second probability), and the probability of exchanging the order is (1 minus the first probability) multiplied by the second probability; the sampling distribution satisfies the coverage condition, and there exists a lower bound α greater than zero that takes the minimum value among the probabilities of discarding the image modality and the temporal modality, the probability of maintaining the normal order, and the probability of exchanging the order, so that the sampling probability of each environment in the finite environment set is not less than α, so that the worst environment risk is constrained by the upper bound of half of the mixed risk.

[0029] In this invention, to ensure the reproducibility of modal adaptive training, the related quantities of mixed risk and worst-case environment risk are defined as follows: Given an input and a target output, the expected negative log-likelihood loss calculated on the input after applying the environmental transformation in a certain environment is called the environmental risk of that environment; the expected value of the environmental risk of the environment according to the sampling distribution q is called the mixed risk, denoted as Rq(θ); the maximum value of the environmental risk in the environment is called the worst-case environment risk, denoted as Rmax(θ) = max Rτ(θ), where τ is taken from the finite set of environments. Modal adaptive training aims to minimize the mixed risk Rq(θ). Under the coverage condition (the sampling probability of each environment is not less than the lower bound α), it can be proved that the worst-case environment risk is constrained by the upper bound of 1 / α of the mixed risk, i.e., Rmax(θ) ≤ (1 / α)·Rq(θ); thus, minimizing the mixed risk can simultaneously constrain the risk under the most unfavorable modal condition, thereby maintaining performance even when either modality is missing, and promoting the consistency of the two single-modal outputs.

[0030] In one implementation, the first probability pd is set to 0.5 and the second probability ps is set to 0.1. The sampling probabilities for the four environments are as follows: for the image-dropping modality and the time-series-dropping modality, the probability is pd / 2 = 0.25; for maintaining the canonical order, it is (1-pd) × (1-ps) = 0.45; and for swapping the order, it is (1-pd) × ps = 0.05. The lower bound of coverage α = min{0.25, 0.45, 0.05} = 0.05. The values ​​of the first probability pd and the second probability ps are both open intervals (0, 1), which can be adjusted according to data and computing power.

[0031] For the protocol-guided modal adaptive ECG world model construction method, the training phase further includes a supervised fine-tuning phase and a reinforcement learning phase, and the step of sampling transformation from a finite environment set and applying it to the model input is performed in both the supervised fine-tuning phase and the reinforcement learning phase; the reward used in the reinforcement learning phase is the sum of format reward (binary), diagnostic accuracy reward and step-by-step evidence reward, wherein the diagnostic accuracy reward is the Jaccard similarity between the predicted diagnostic label set and the ground truth diagnostic label set, and the step-by-step evidence reward is the average coverage of the key diagnostic evidence phrase (i.e., the evidentiary description of structured features) corresponding to each step in the response.

[0032] The present invention provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the protocol-guided modal adaptive electrocardiogram world model construction method provided by the present invention.

[0033] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the protocol-guided modal adaptive electrocardiogram world model construction method provided by the present invention.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] This invention provides a protocol-guided modal adaptive ECG world model construction method. First, a world model representing cardiac electrical activity is constructed from unlabeled ECGs using a self-supervised approach, resulting in a pre-trained temporal encoder that has learned morphological and rhythmic representations. Then, based on this encoder, combined with protocol-guided data generation and modal adaptive training, a protocol-guided, modal adaptive ECG world model is constructed. This constructed world model can be used for low-illusion, grounded, structured interpretation of input ECG outputs. The technical advantages of this invention include:

[0036] First, a pre-trained temporal encoder was constructed from the perspective of heartbeat as words and rhythm as sentences. It learned morphological and rhythmic general representations from unlabeled data and constructed a significantly expanded electrocardiogram vocabulary, overcoming the shortcomings of fixed window segmentation that ignores morphology, rhythm and latent semantics.

[0037] Secondly, by using protocols and quantitative thresholds to constrain the interpretation of the generated data, the illusion of corpus and factual errors are significantly reduced, and the completeness of the interpretation and the grounding of the evidence are improved.

[0038] Third, in the construction of the ECG world model and modal adaptive training, decoupled modal encoding and interleaved modal dropping provide explicit theoretical guarantees of modal robustness and cross-modal consistency without introducing additional missing modal generators and explicit alignment losses, so that the model can still maintain high diagnostic performance when any modality is missing. Attached Figure Description

[0039] Figure 1 This is a flowchart of the method proposed in this invention;

[0040] Figure 2 This is a flowchart of the pre-trained temporal encoder constructed in this invention;

[0041] Figure 3 This is a schematic diagram comparing the perspective of using heartbeat as a word and rhythm as a sentence in this invention with the traditional fixed-window segmentation perspective;

[0042] The left image shows the traditional fixed window perspective; the right image shows the perspective of heartbeat as words and rhythm as sentences.

[0043] Figure 4 A structural block diagram of the timing encoder backbone network constructed for this invention;

[0044] Figure 5This is a flowchart illustrating the process of generating protocol guidance data for this invention.

[0045] Figure 6 This is a schematic diagram of the decoupled bimodal architecture and interleaved modality discarding in the ECG world model construction and modal adaptive training of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.

[0047] This invention provides a self-supervised representation learning and multimodal modeling method for electrocardiograms (ECGs), including a method for constructing a protocol-guided, modally adaptive ECG (cardiac electrical activity) world model, and an electronic device and computer-readable storage medium carrying the method. To facilitate understanding of the logical relationships between the steps, a summary is provided first: This invention first undergoes a pre-training step, training a pre-trained temporal encoder in a self-supervised manner. This encoder serves as the foundation (backbone network) of the ECG world model; at this stage, a complete ECG world model is not yet obtained. Subsequently, in the model construction step, this pre-trained temporal encoder is used as the temporal encoder to build the ECG world model. Then, in the modal adaptive training step, interleaved modal dropout training is applied to the ECG model, ultimately obtaining a modally adaptive ECG world model. In other words, "first, a foundation encoder is pre-trained, then a complete ECG world model is constructed and trained." Figure 1 As shown, the method of the present invention includes the following steps in sequence:

[0048] The first step is the construction of a pre-trained temporal encoder:

[0049] like Figure 2 As shown, this step includes three stages: heartbeat word segmentation to generate ECG sentences, morphological and rhythmic representation learning, and masked sentence pre-training. The heartbeat word segmenter detects QRS waves (QRS waves are a combination of waveforms representing rapid ventricular depolarization in an ECG, consisting of a Q wave (initially downward), an R wave (upward spike), and an S wave (downward fall), reflecting the process of electrical signals being conducted from the atria to the ventricles and triggering contraction), segments the heartbeat into ECG words, and splices them into ECG sentences by lead. An ECG word list is constructed through heartbeat vector quantization and reconstruction. Then, partial ECG words in the ECG sentences are masked, and their corresponding set of ECG words is predicted based on context, thus obtaining a self-supervised pre-trained temporal encoder.

[0050] like Figure 3As shown, unlike the traditional approach of dividing the electrocardiogram into fixed-size, fixed-step time windows, this invention divides the signal into electrocardiogram words (equivalent to "words") based on the midpoint of adjacent QRS wave indices, and regards the sequence of multiple heartbeats as rhythms (equivalent to "sentences"), thereby preserving the morphological characteristics of heartbeats and the rhythmic semantics between heartbeats.

[0051] The second step is the protocol bootstrap data generation process:

[0052] like Figure 5 As shown, a deterministic feature extractor extracts structured physiological features from the signal, one lead at a time and one heartbeat at a time. Based on a multi-stage interpretation protocol with quantitative thresholds, protocol-aware prompts are assembled to guide the interpretation generation model to produce structured interpretation corpus containing multi-step reasoning trajectories and final diagnosis.

[0053] The third step is the construction of the ECG world model:

[0054] First, a pre-trained temporal encoder (i.e., backbone / base) is obtained through self-supervised training. Then, a decoupled bimodal ECG world model is built using this pre-trained temporal encoder. Subsequently, intermodal dropout training is applied to the ECG world model to obtain the final trained ECG world model.

[0055] like Figure 6 As shown, a multimodal ECG model is constructed with independent temporal encoders, image encoders and their own independent projection layers, and with temporal tokens injected by explicit ECG labels. The temporal encoder is a pre-trained encoder. The model sets explicit ECG labels and injects the temporal tokens obtained by the temporal encoder only at the ECG labels.

[0056] The fourth step is the modality adaptive training process:

[0057] Using the aforementioned corpus, a sampling transformation induced by a first probability and a second probability is applied to the input from a finite set of environments containing modality dropout and order exchange, to minimize the cross-environment mixed risk optimization parameters, resulting in a modality-adaptive ECG world model. This constructed ECG world model is used for the structured interpretation of the input ECG output, which includes multi-step inference trajectories and the final diagnosis. Figure 6 As shown, interleaved modalities are discarded from the inputs applied to the ECG world model during the training phase.

[0058] To ensure the reproducibility of modal adaptive training, this invention defines the related quantities of mixed risk and worst-case environment risk as follows: Given an input and a target output, the expected negative log-likelihood loss calculated for the input after applying the environmental transformation in a certain environment is called the environmental risk of that environment; the expected value of the environmental risk of the environment according to the sampling distribution q is called the mixed risk, denoted as Rq(θ); the maximum value of the environmental risk in the environment is called the worst-case environment risk, denoted as Rmax(θ) = max Rτ(θ), where τ is taken from the finite set of environments. Modal adaptive training aims to minimize the mixed risk Rq(θ). Under the coverage condition (the sampling probability of each environment is not less than the lower bound α), it can be proved that the worst-case environment risk is constrained by the upper bound of 1 / α of the mixed risk, i.e., Rmax(θ) ≤ (1 / α)·Rq(θ); thus, minimizing the mixed risk can simultaneously constrain the risk under the most unfavorable modal condition, thereby maintaining performance even when either modality is missing and promoting the consistency of the two single-modal outputs.

[0059] In one implementation, the first probability pd is set to 0.5 and the second probability ps is set to 0.1. The sampling probabilities for the four environments are as follows: for the image-dropping modality and the time-series-dropping modality, the probability is pd / 2 = 0.25; for maintaining the canonical order, it is (1-pd) × (1-ps) = 0.45; and for swapping the order, it is (1-pd) × ps = 0.05. The lower bound of coverage α = min{0.25, 0.45, 0.05} = 0.05. The values ​​of the first probability pd and the second probability ps are both open intervals (0, 1), which can be adjusted according to data and computing power.

[0060] Preferably, in the protocol-guided data generation step, the multi-stage interpretation protocol is reorganized from the clinical monograph process into five stages: technical quality control-heart rate-rhythm, conduction-electric axis-interval, cardiac chamber hypertrophy-voltage, ischemia-infarction-differentiation, and electrolyte-QT, and is forced to exclude differential diagnoses by explicit negative findings; the sampling distribution satisfies the coverage condition and has a lower bound α greater than zero, so that the worst-case risk (the risk corresponding to the highest risk in the four environments, i.e., the maximum value of each environment risk) is constrained by an upper bound of 1 / α of the mixed risk; the training stage includes two stages: supervised fine-tuning and reinforcement learning, both of which apply sampling transformations, and the reward for the reinforcement learning stage is the sum of format reward, diagnostic accuracy reward, and step-by-step evidence reward.

[0061] like Figure 1 As shown, in specific implementation, the protocol-guided modal adaptive ECG world model construction method provided by this invention includes:

[0062] 1) Steps for building a pre-trained temporal encoder:

[0063] The system detects QRS waves from the raw electrocardiogram (ECG) signal using a heartbeat segmentation tool and segments the heartbeats into ECG words. These ECG words from multiple leads are then sequentially concatenated to form an ECG sentence. Vector quantization and reconstruction of the heartbeats are used to construct an ECG vocabulary consisting of several discrete sets of ECG words. Parts of the ECG words in the ECG sentence are masked, and the set of ECG words corresponding to the masked parts is predicted based on the unmasked context. This self-supervised learning of the morphological and rhythmic representations of the ECG signal yields a pre-trained ECG timing encoder.

[0064] 2) Protocol bootstrapping data generation steps:

[0065] Structured physiological features per lead and per heartbeat are extracted from the raw electrocardiogram (ECG) signal using a deterministic feature extractor to obtain a feature dictionary. Based on a pre-constructed multi-stage ECG interpretation protocol containing quantitative thresholds and systematic diagnostic logic, and the feature dictionary, a protocol-based perception prompt is assembled. This protocol-based perception prompt is input into the interpretation generation model to generate an ECG interpretation following a fixed template. This fixed template includes a thought chain portion encoding the multi-step protocol reasoning trajectory, a summary portion, and an answer portion providing the final diagnosis. The above protocol-guided data generation process is repeated to obtain the interpretation corpus.

[0066] 3) Model building steps:

[0067] An electrocardiogram (ECG) world model is constructed, which has independent temporal encoders and image encoders, as well as independent projection layers. The temporal encoder is the pre-trained ECG temporal encoder. The model sets explicit ECG labels and injects temporal tokens obtained by the temporal encoder only at the ECG labels.

[0068] 4) Modality adaptive training steps:

[0069] Using the aforementioned interpretation corpus, during the training phase, a sampling transformation is applied to the model input from a finite set of environments according to a sampling distribution. This finite set of environments includes four types of environments: discarding image modalities, discarding temporal modalities, maintaining the canonical order of modal token blocks, and swapping the order of modal token blocks. The sampling distribution is induced by two independent trials: a modality discarding trial with a first probability and a token order swapping trial with a second probability while retaining two modalities. The model parameters are optimized with the goal of minimizing the cross-environment mixing risk under the sampling distribution, resulting in a modality-adaptive ECG world model.

[0070] The ECG world model is used to provide a structured interpretation of the input ECG output, including multi-step reasoning trajectories and a final diagnosis.

[0071] In specific implementation, further, the heartbeat is segmented into ECG terms by: bandpass filtering the signal of a lead and then integrating it with a moving wave to obtain an integrated signal; traversing the local maxima of the integrated signal, and determining the local maxima that are after the refractory period and exceed the detection threshold as QRS waves; for each lead, using the midpoint of the adjacent QRS wave index as the interval boundary to segment the ECG term corresponding to the heartbeat, and padding ECG terms that are less than the set time window length with zeros; and concatenating the ECG terms of multiple leads in sequence to form an ECG sentence, and unifying the length of the ECG sentence to the set length, padding those that are insufficient with zeros and truncating those that exceed the set length.

[0072] Constructing an electrocardiogram (ECG) vocabulary includes: obtaining several representations from ECG sentences using the time-series encoder; quantizing each representation by finding the nearest neighbor set of ECG words in the ECG vocabulary using cosine similarity; reconstructing the original heartbeat signal using the quantized set of ECG words via a decoder; jointly optimizing the quantization and reconstruction processes using the reconstruction error and the loss consisting of the distance between the representation and the set of ECG words; and updating the ECG vocabulary using an exponential moving average strategy.

[0073] like Figure 4 As shown, the temporal encoder includes: a lexical embedding module, spatial embedding, temporal embedding, positional embedding, category lexical units, and a transformer encoder; predicting the set of ECG words corresponding to the masked part based on the unmasked context includes: randomly masking the ECG words in the ECG sentence according to a set masking ratio, replacing the masked ECG words with learnable masking labels, inputting the hidden vector obtained by the temporal encoder into a linear classifier, predicting the index of the set of ECG words corresponding to the masked ECG words, and using the difference between the predicted index and the true index as the training target; the temporal encoder maps each ECG word to a lexical unit through one-dimensional convolution, and then inputs it into the transformer encoder after superimposing spatial embedding, temporal embedding, and positional embedding, wherein the spatial embedding maps each ECG word back to its lead, and the temporal embedding maps each ECG word to the time interval of its QRS wave.

[0074] The structured physiological features include heart rate, RR interval, P wave amplitude and duration, PR interval, QRS amplitude and duration, T wave amplitude and duration, ST segment descriptor, and QT / QTc interval. For each of the twelve leads, multiple time-ordered feature sequences are constructed. The multi-stage ECG interpretation protocol is reorganized from the ECG interpretation process in clinical medical monographs into five stages: technical quality control-heart rate-rhythm stage, conduction-electric axis-interval stage, cardiac chamber hypertrophy-voltage stage, ischemia-infarction-differentiation stage, and electrolyte-QT stage.

[0075] The protocol-aware prompts include differential exclusion instructions to exclude differential diagnoses with explicit negative findings. The quantitative thresholds include at least one of the following: left ventricular hypertrophy voltage standard, normal PR interval range, QRS duration threshold, ST segment elevation threshold, pathological Q wave standard, and corrected upper limit of QT interval. The multi-step protocol reasoning trajectory consists of six steps, including a five-step analysis corresponding to the five stages and a sixth step of final medical reasoning. The process of generating ECG interpretation adopts a blind interpretation simulation method, that is, interpretation is performed without explicitly mentioning the existence of the reference report, and the reasoning of the interpretation is aligned with the internal structure of the ECG report used as the basis for the answer, thereby identifying unmarked abnormalities while conforming to the ECG report.

[0076] The sampling probabilities of the four environments satisfy the following: the probability of discarding the image mode and the probability of discarding the temporal mode are both half of the first probability; the probability of maintaining the normal order is (1 minus the first probability) multiplied by (1 minus the second probability); and the probability of exchanging the order is (1 minus the first probability) multiplied by the second probability. The sampling distribution satisfies the coverage condition, and there exists a lower bound α greater than zero that takes the minimum value among the probabilities of discarding the image mode and the temporal mode, maintaining the normal order, and exchanging the order, so that the sampling probability of each environment in the finite environment set is not less than α, and thus the worst-case environment risk is constrained by the upper bound of half of the mixed risk.

[0077] The training phase includes a supervised fine-tuning phase and a reinforcement learning phase, and both the supervised fine-tuning phase and the reinforcement learning phase perform the step of sampling transformation from a finite set of environments and applying it to the model input; the reward used in the reinforcement learning phase is the sum of a format reward, a diagnostic accuracy reward, and a step-by-step evidence reward, wherein the diagnostic accuracy reward is the Jaccard similarity between the predicted diagnostic label set and the ground truth diagnostic label set, and the step-by-step evidence reward is the average coverage of the key diagnostic evidence phrase corresponding to each step in the response.

[0078] This invention provides an electronic device, including a processor and a memory. The memory stores a computer program, and when the processor executes the computer program, it implements the protocol-guided modal adaptive electrocardiogram (ECG) world model construction method provided by this invention. This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the protocol-guided modal adaptive ECG world model construction method provided by this invention.

[0079] In summary, this invention provides a protocol-guided modality-adaptive ECG world model construction method, treating the ECG as a language with heartbeats as words and rhythms as sentences: A heartbeat segmenter segments the original signal into heartbeats and concatenates them into ECG sentences. An ECG vocabulary is constructed through vector quantization reconstruction, and self-supervised pre-training is performed using masked sentence prediction to obtain a pre-trained temporal encoder that has learned morphological and rhythmic representations. Based on this, a low-illusion, structured interpretation corpus is generated through protocol guidance. A decoupled bimodal architecture using this pre-trained encoder as the temporal encoder is employed, and interleaved modality dropout is applied during training to minimize cross-environment mixing risk, thereby constructing a protocol-guided, modality-adaptive ECG world model. The constructed ECG world model can provide a structured interpretation of the input ECG output, including the reasoning process and the final diagnosis. This invention significantly reduces interpretation illusions and maintains high diagnostic performance and cross-modal consistency even when any modality of the ECG signal or image is missing.

[0080] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A protocol-guided modal adaptive electrocardiogram world model construction method, characterized in that, Includes the following steps: 1) Construct a pre-trained temporal encoder: 11) Detect QRS waves from the raw electrocardiogram signal and segment the heartbeat into electrocardiogram words, then concatenate the electrocardiogram words from multiple leads in sequence to form an electrocardiogram sentence; 12) Construct an electrocardiogram vocabulary by vector quantization and reconstruction of heartbeats; The electrocardiogram (ECG) vocabulary consists of discrete sets of ECG terms; 13) Mask some ECG words in the ECG sentence and predict the set of ECG words corresponding to the masked part based on the unmasked context. Learn the morphological and rhythmic representations of the ECG signal in a self-supervised manner to obtain the pre-trained temporal encoder, that is, obtain the model backbone through self-supervised pre-training. 2) Construct a multi-stage ECG interpretation protocol and generate protocol guidance data for the ECG world model to interpret ECGs by executing the protocol: 21) Extract structured physiological features for each lead and each heartbeat from the original electrocardiogram signal to obtain a feature dictionary; 22) Based on a pre-constructed multi-stage electrocardiogram interpretation protocol that includes quantitative thresholds and systematic diagnostic logic, protocol-aware prompts are obtained by assembling with a feature dictionary; 23) Generate an ECG interpretation following a fixed template based on protocol-aware prompts; the fixed template includes a thought chain part that encodes a multi-step protocol reasoning trajectory, a summary part, and an answer part that gives the final diagnosis; Repeating the above process yields the corpus for interpretation; 3) Construct an electrocardiogram (ECG) world model; the ECG world model is equipped with a timing encoder; The timing encoder is the pre-trained timing encoder constructed in step 1); the ECG world model sets explicit ECG labels and injects timing tokens obtained by the timing encoder only at the ECG labels; 4) Modal adaptive training of the ECG world model: The training phase employs a corpus that is interpreted and then sampled and transformed from a finite set of environments according to a sampling distribution, and applied to the model input. The finite set of environments includes environments that discard image modalities, environments that discard temporal modalities, environments that maintain the canonical order of modal token blocks, and environments that swap the order of modal token blocks. The sampling distribution is induced by modal drop trials with modal drop probability (i.e., the first probability) and token order swap trials with token order swap probability (i.e., the second probability) while retaining both modalities; The model parameters are optimized with the goal of minimizing the cross-environmental mixing risk under the sampling distribution, thus obtaining a modally adaptively trained ECG world model; The trained ECG world model is used to provide a structured interpretation of the input ECG output, including multi-step reasoning trajectories and the final diagnosis.

2. The protocol-guided modal adaptive electrocardiogram world model construction method as described in claim 1, characterized in that, In step 11), specifically, the QRS wave is detected from the original electrocardiogram signal by the heartbeat word segmenter and the heartbeat is segmented into electrocardiogram words. The electrocardiogram words of multiple leads are then spliced ​​together in sequence to form an electrocardiogram sentence.

3. The protocol-guided modal adaptive electrocardiogram world model construction method as described in claim 2, characterized in that, The heartbeat is divided into electrocardiogram terms, specifically including: After bandpass filtering the signal from one lead, the integrated signal is obtained by moving wave integration; The local maxima of the integral signal are traversed, and the local maxima that are after the refractory period and exceed the detection threshold are identified as QRS waves. For each lead, the ECG word corresponding to the heartbeat is obtained by dividing the interval using the midpoint of the adjacent QRS wave index as the interval boundary. ECG words that are less than the set time window length are filled with zeros. The ECG words from multiple leads are sequentially concatenated to form an ECG sentence, and the length of the ECG sentence is standardized to a set length. If the length is insufficient, it is padded with zeros, and if it exceeds the set length, it is truncated. In this way, the heartbeat is divided into ECG words.

4. The method for constructing a protocol-guided modal adaptive electrocardiogram world model as described in claim 1, characterized in that, Step 12) Constructing the electrocardiogram vocabulary includes: The electrocardiogram sentence is processed by a time-series encoder to obtain multiple representations; For each representation, the set of ECG words whose nearest neighbors are found using cosine similarity is quantified; Reconstruct the original heartbeat signal based on the quantized set of ECG words; The quantization and reconstruction processes are jointly optimized using the reconstruction error and the loss consisting of the distance between the characterization and the set of ECG terms, and the ECG term list is updated using an exponential moving average strategy.

5. The method for constructing a protocol-guided modal adaptive electrocardiogram world model as described in claim 1, characterized in that, In step 13), predicting the set of ECG words corresponding to the masked portion based on the unmasked context specifically includes: The ECG words in the ECG sentence are randomly masked according to a set masking ratio, and the masked ECG words are replaced with learnable masking tags; The electrocardiogram representation obtained by the pre-trained temporal encoder is input into the linear classifier to predict the index of the corresponding set of electrocardiogram words; The difference between the predicted index and the true index is used as the training objective; Each ECG word is mapped to a word through one-dimensional convolution, and then spatial embedding, temporal embedding, and positional embedding are superimposed before being input into a timing encoder. Among them, spatial embedding maps each ECG word back to its lead, and temporal embedding maps each ECG word to the time interval of its QRS wave.

6. The method for constructing a protocol-guided modal adaptive electrocardiogram world model as described in claim 1, characterized in that, The multi-stage ECG interpretation protocol specifically reorganizes the ECG interpretation process into five stages: technical quality control-heart rate-rhythm stage, conduction-electric axis-interval stage, cardiac hypertrophy-voltage stage, ischemia-infarction-differentiation stage, and electrolyte-QT stage. The protocol's sensing prompts include differential exclusion instructions to exclude differential diagnoses with explicit negative findings. The quantitative thresholds include at least one of the following: left ventricular hypertrophy voltage standard, normal PR interval range, QRS duration threshold, ST segment elevation threshold, pathological Q wave standard, and upper limit of corrected QT interval.

7. The protocol-guided modal adaptive electrocardiogram world model construction method as described in claim 6, characterized in that, The multi-step protocol reasoning trajectory includes analysis corresponding to the five stages and final medical reasoning. The process of generating ECG interpretation adopts a blind interpretation simulation method, that is, interpretation is carried out without explicitly mentioning the existence of the reference report and the reasoning of the interpretation is aligned with the internal ECG report used as the basis for the answer, thereby identifying unlabeled abnormalities while conforming to the ECG report.

8. The method for constructing a protocol-guided modal adaptive electrocardiogram world model as described in claim 1, characterized in that, The environment sampling probability is induced by the first probability and the second probability. In modal adaptive training, this ensures that the model is fully trained under various modal input conditions. The sampling probability satisfies the following: the probability of discarding the image modality and the probability of discarding the temporal modality are both half of the first probability, and the probability of maintaining the normal order and the probability of exchanging the order are determined. The sampling distribution satisfies the coverage condition and has a lower bound greater than zero, such that the sampling probability of each environment in the finite set of environments is not less than this lower bound.

9. The method for constructing a protocol-guided modal adaptive electrocardiogram world model as described in claim 1, characterized in that, In step 4), the training phase includes a supervised fine-tuning phase and a reinforcement learning phase, and in both the supervised fine-tuning phase and the reinforcement learning phase, a sampling transformation from a finite set of environments is performed and applied to the model input; The reward used in the reinforcement learning phase is the sum of format reward, diagnostic accuracy reward, and step-by-step evidence reward; where diagnostic accuracy reward is the Jaccard similarity between the predicted diagnostic label set and the ground truth diagnostic label set; and step-by-step evidence reward is the average coverage of the corresponding key diagnostic evidence phrases in the response.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the protocol-guided modal adaptive electrocardiogram world model construction method according to any one of claims 1 to 10.

Images