A supervised pre-training based multi-modal electrocardiosignal representation learning method

Abstract

This application relates to a supervised pre-training-based multimodal electrocardiogram (ECG) signal representation learning method. The method includes: entity extraction and standardized mapping of clinical text reports to obtain structured diagnostic labels; extraction of ECG features from raw ECG data through cross-channel slicing and routing aggregation using a multi-granularity ECG encoder; inputting the structured diagnostic labels and ECG features into a multimodal fusion network to complete text semantic extraction and cross-modal interaction, obtaining fused features; constructing modality consistency loss and classification loss to jointly optimize model parameters; and inputting the ECG data to be tested into the optimized model to complete diagnostic prediction. This method can fully exploit the value of clinical text, achieve fine-grained alignment of cross-modal features, reduce the computational complexity of long sequences, and take into account the multi-scale features of ECG signals, effectively improving the accuracy, robustness, and generalization ability of ECG analysis models.

Description

Technical Field

[0001] This invention belongs to the field of signal processing and cardiac disease diagnosis technology, and in particular relates to a method for learning multimodal electrocardiogram signal representation based on supervised pre-training. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep learning has been widely applied in the field of automatic electrocardiogram (ECG) signal analysis. ECG analysis models based on convolutional neural networks and Transformer architectures can automatically extract features from raw signals, enabling auxiliary diagnosis of various cardiac diseases such as arrhythmias and myocardial ischemia. These methods typically employ a supervised learning paradigm, using a large amount of labeled ECG data to train classification models.

[0003] Traditional electrocardiogram (ECG) signal analysis methods are mainly divided into two categories: one is machine learning methods based on handcrafted features, which classify ECGs by extracting statistical features such as heart rate variability and waveform morphology; the other is end-to-end learning methods based on deep neural networks, which directly map the raw ECG signal to disease categories. In recent years, Transformer-based models have been introduced into the field of ECG analysis due to their powerful long-term modeling capabilities, capturing global dependencies in ECG signals through a self-attention mechanism.

[0004] However, existing technologies still face the following technical challenges in practical applications: First, supervised learning relies on a large amount of high-quality labeled data, while ECG reports, as text records generated in clinical practice, contain rich diagnostic information that has not been effectively utilized. Directly using the original reports also presents problems such as inconsistent terminology and noise interference. Second, single-modal ECG analysis models only focus on the temporal characteristics of the signal itself, ignoring the semantic information contained in the clinical text description, resulting in insufficient understanding of the pathological features of the ECG signal. Third, traditional multimodal methods typically employ simple approaches such as feature concatenation or fractional fusion, lacking a mechanism for fine-grained alignment of ECG signal representation with text semantics in the feature space, thus limiting the complementary enhancement effect of cross-modal information. Fourth, existing Transformer models have high computational complexity when processing high-sampling-rate, long-term ECG signals, and single-scale slices are difficult to simultaneously capture local waveform details and global rhythm patterns. Summary of the Invention

[0005] Therefore, it is necessary to provide a method for learning multimodal ECG signal representation based on supervised pre-training to address the aforementioned technical problems.

[0006] Firstly, this application provides a method for learning multimodal electrocardiogram signal representations based on supervised pre-training, including:

[0007] S1. Obtain clinical text reports, extract entities from the clinical text reports and perform standardized mapping to obtain structured diagnostic label data; among which, the structured diagnostic label data is used to characterize the diagnostic categories of heart diseases;

[0008] S2. Obtain raw electrocardiogram (ECG) data. Input the raw ECG data into a multi-granularity ECG encoder for cross-channel slicing and routing aggregation to obtain ECG feature data. The ECG feature data is used to characterize the spatiotemporal features of ECG signals at multiple scales.

[0009] S3. Input the structured diagnostic label data and ECG feature data into a multimodal fusion network for text semantic extraction and cross-modal interaction to obtain text semantic features and fusion features; among them, the fusion features are used to characterize the pathological features of ECG signals under text semantic constraints.

[0010] S4. Based on ECG feature data, text semantic features, and fusion features, modality consistency loss and classification loss are constructed. Based on modality consistency loss and classification loss, the multi-granularity ECG encoder and multi-modal fusion network are jointly optimized to obtain the ECG analysis model.

[0011] S5. Input the ECG data to be tested into the ECG analysis model for feature encoding and classification prediction to obtain the ECG analysis results.

[0012] In one embodiment, S1 includes:

[0013] S11. Construct prompts containing knowledge of the cardiology field based on the standard ECG terminology database and common clinical abbreviations. Then, concatenate the prompts and clinical text reports and input them into the large language model for fine-tuning of the prompts to parse the clinical text reports and output structured text containing diagnostic entity names and deterministic values.

[0014] S12. Filter out diagnostic entities in the structured text whose deterministic values ​​are lower than a preset threshold to obtain a set of high-confidence diagnostic entities.

[0015] S13. Input each diagnostic entity in the high-confidence diagnostic entity set into the pre-trained medical text encoder for semantic encoding to obtain entity semantic vectors, and input each standard term in the standard ECG terminology library into the medical text encoder for semantic encoding to obtain a standard semantic vector library.

[0016] S14. Calculate the cosine similarity between the semantic vector of each entity and all vectors in the standard semantic vector library. Select the standard term corresponding to the maximum cosine similarity as the standardized mapping result of the diagnostic entity. Keep the diagnostic entities with cosine similarity lower than the preset matching threshold as new temporary terms. Merge and deduplicate the diagnostic entities to generate structured diagnostic label data.

[0017] In one embodiment, S2 includes:

[0018] S21. Preset at least two sets of time windows of different sizes as slice granularity. For each slice granularity, construct a three-dimensional tensor slice from all leads of ECG data points within the same time window. Map the three-dimensional tensor slice to a one-dimensional feature vector through a learnable linear projection layer to obtain the slice feature sequence at each granularity. Add independent learnable positional encoding to the slice feature sequence at each granularity.

[0019] S22. Input the slice feature sequence at each granularity into the multi-layer Transformer module for self-attention calculation. During the calculation process of the multi-layer Transformer module, extract the first output feature of the shallow network layer, the second output feature of the middle network layer, and the third output feature of the deep network layer. After concatenating the first output feature, the second output feature, and the third output feature, perform dimensionality reduction mapping through the first fully connected projection layer to obtain the enhanced slice feature sequence.

[0020] S23. Initialize learnable routing tokens for each granularity. The learnable routing token is a trainable parameter vector with the same feature dimension as the slice feature sequence. For each granularity, use the learnable routing token as the query matrix and the enhanced slice feature sequence under the granularity as the key matrix and value matrix. Calculate the attention weights through a multi-head cross-attention mechanism, and aggregate the enhanced slice feature sequence into the learnable routing token according to the attention weights to output the routing token of the granularity.

[0021] S24. Stack the routing tokens corresponding to at least two granularities in the sequence dimension to generate a token sequence. Input the token sequence into the Transformer encoder layer for self-attention calculation. After the self-attention calculation, concatenate all the routing tokens of all granularities output in the feature dimension and then fuse and transform them through the second fully connected projection layer to obtain ECG feature data.

[0022] In one embodiment, S3 includes:

[0023] S31. Perform word segmentation on the diagnostic label text in the structured diagnostic label data to obtain a word index sequence, and convert the word index sequence into a word vector sequence through a word embedding layer;

[0024] S32. Input the word vector sequence into the Transformer encoder of the pre-trained language model to perform context modeling, obtain the hidden state vector corresponding to each word, and extract the output vector corresponding to the special marker as the semantic representation of the entire diagnostic label text.

[0025] S33. Perform dimensional transformation on the semantic representation to obtain the text semantic features.

[0026] In one embodiment, S3 further includes:

[0027] S41. Input the ECG feature data into the linear extension layer for transformation to obtain the ECG feature sequence, and use the ECG feature sequence as the encoder memory in the Transformer decoder architecture.

[0028] S42. Use the text semantic features as the initial input query vector of the decoder in the Transformer decoder architecture. In each layer of the Transformer decoder, use the decoder input query vector as the query matrix and the ECG feature sequence as the key matrix and value matrix. Calculate the interaction weight between the text semantic features and the ECG feature sequence. Based on the interaction weight, dynamically aggregate the ECG feature sequence and the text semantic features to generate the decoder output of the current layer.

[0029] S43. The feature vector output by the last decoder layer is used as the fusion feature, and the fusion feature is input into the classification head composed of fully connected layers to obtain the classification prediction result.

[0030] In one embodiment, S4 includes:

[0031] S51. Perform L2 norm normalization on the ECG feature data and text semantic features respectively to obtain normalized ECG features and normalized text features.

[0032] S52. In a training batch, calculate the cosine similarity between all normalized ECG features and all normalized text features to obtain the cosine similarity matrix. The expression for cosine similarity is:

[0033]

[0034] in, Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of text features of each sample. Indicates the first in the batch ECG characteristic data of each sample Indicates the first in the batch Textual semantic features of each sample Represents L2 norm operations;

[0035] S53. Construct a multi-label vector for each sample within a batch based on the structured diagnostic label data. Calculate the label similarity between any two sample multi-label vectors to obtain a label similarity matrix. Set a positive sample threshold and a negative sample threshold. Map the positions in the label similarity matrix that are greater than the positive sample threshold to the same positions in the cosine similarity matrix and assign them a value of 1 to construct a positive sample mask matrix. Map the positions in the label similarity matrix that are less than the negative sample threshold to the same positions in the cosine similarity matrix and assign them a value of 1 to construct a negative sample mask matrix.

[0036] S54. Based on the positive sample mask matrix, the positive sample loss is calculated. The expression for the positive sample loss is:

[0037]

[0038] in, Indicates positive sample loss. Indicates batch size, Represents the positive sample mask matrix. Line 1 Column elements, This represents the sum of all elements in the positive sample mask matrix. Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of textual features among the samples;

[0039] S55. Based on the negative sample mask matrix, the negative sample loss is calculated. The expression for the negative sample loss is:

[0040]

[0041] in, Indicates negative sample loss. Indicates batch size, Represents the positive sample mask matrix. Line 1 Column elements, This represents the sum of all elements in the positive sample mask matrix. Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of text features of each sample. This represents the preset boundary value hyperparameters;

[0042] S56. The modality consistency loss is obtained by weighted summing of the positive sample loss and the negative sample loss. The expression for the modality consistency loss is:

[0043]

[0044] in, This represents the modal consistency loss. Indicates positive sample loss. Indicates negative sample loss. These are the weighting coefficients for the positive sample loss. The weighting coefficients are for the negative sample loss.

[0045] In one embodiment, S4 further includes:

[0046] S61. Based on the fusion features, the existence probability of each preset diagnostic category is obtained through the fully connected layer of the classification head and the sigmoid activation function;

[0047] S62. Construct a multi-label true value vector based on the structured diagnostic label data;

[0048] S63. Calculate the binary cross-entropy loss as the classification loss. The expression for the classification loss is:

[0049]

[0050] in, Represents classification loss, Represents a multi-labeled true value vector. Indicates the first In the nth sample The probability of the existence of each diagnostic category For batch size, This represents the total number of preset diagnostic categories.

[0051] Secondly, this application also provides a supervised pre-training-based multimodal electrocardiogram signal representation learning system, comprising:

[0052] The text data acquisition and standardization module is used to acquire clinical text reports, extract entities from the clinical text reports, and perform standardized mapping to obtain structured diagnostic label data; among which, the structured diagnostic label data is used to characterize the diagnostic categories of heart diseases;

[0053] The ECG data processing module is used to acquire raw ECG data, input the raw ECG data into a multi-granularity ECG encoder for cross-channel slicing and routing aggregation, and obtain ECG feature data; among which, the ECG feature data is used to characterize the spatiotemporal features of ECG signals at multiple scales;

[0054] The data fusion and analysis module is used to input structured diagnostic label data and electrocardiogram (ECG) feature data into a multimodal fusion network for text semantic extraction and cross-modal interaction, resulting in text semantic features and fusion features. Among them, the fusion features are used to characterize the pathological features of ECG signals under text semantic constraints.

[0055] The ECG analysis model training module is used to construct modality consistency loss and classification loss based on ECG feature data, text semantic features and fusion features, and then jointly optimize the multi-granularity ECG encoder and multi-modal fusion network based on the modality consistency loss and classification loss to obtain the ECG analysis model.

[0056] The ECG data analysis module is used to input the ECG data to be tested into the ECG analysis model for feature encoding and classification prediction to obtain the ECG analysis results.

[0057] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0058] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0059] The aforementioned supervised pre-training-based multimodal ECG signal representation learning method fully mines and utilizes diagnostic information from clinical text reports. Through entity extraction and standardized mapping, it eliminates inconsistencies in terminology and noise interference within the reports, transforming unstructured text into structured supervised signals and overcoming the reliance on large amounts of manually labeled data for supervised learning. Multi-granularity ECG coding is used to comprehensively capture multi-scale spatiotemporal features of local waveforms and global rhythms in ECG signals. The routing aggregation mechanism significantly reduces the computational complexity of long-series signals. Combined with a multimodal fusion network, it achieves fine-grained cross-modal interaction between text semantics and ECG features. Modal consistency loss further enhances the cross-modal feature alignment effect, compensating for the shortcomings of insufficient pathological understanding in single-modal models and poor complementary enhancement effects in traditional multimodal fusion. This improves the accuracy and robustness of ECG analysis model diagnosis, enhances the model's generalization ability, and provides reliable support for the auxiliary diagnosis of clinical cardiac diseases. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart illustrating a multimodal electrocardiogram signal representation learning method based on supervised pre-training in one embodiment;

[0062] Figure 2This is a flowchart illustrating a supervised pre-training-based multimodal ECG signal representation learning method in one embodiment;

[0063] Figure 3 This is a flowchart illustrating the structure of an electrocardiogram feature encoder in one embodiment;

[0064] Figure 4 This is a flowchart illustrating the modal consistency loss module in one embodiment;

[0065] Figure 5 This is a schematic diagram of the structure of a multimodal electrocardiogram signal representation learning system in one embodiment;

[0066] Figure 6 This is a schematic diagram of the structure of a multimodal electrocardiogram signal representation learning system based on supervised pre-training in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] In one embodiment, reference Figure 1 The document presents a flowchart illustrating the supervised pre-training-based multimodal electrocardiogram (ECG) signal representation learning method provided in this application. This embodiment uses the application of this method to an ECG analysis terminal (hereinafter referred to as the terminal) as an example. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0069] S1. Obtain clinical text reports, extract entities from the clinical text reports and perform standardized mapping to obtain structured diagnostic label data.

[0070] For example, the ECG analysis terminal obtains a clinical text report from the electronic medical record system (EMRS) of a medical institution. This report is unstructured text generated by clinicians based on ECG diagnosis, containing core information such as descriptions of heart disease diagnoses and waveform abnormality assessments. The ECG analysis terminal performs diagnostic entity extraction and standardization mapping operations on the clinical text report to transform the unstructured text into structured and standardized diagnostic label data, which is used to represent the diagnostic categories of heart diseases. The extraction process relies on entity recognition technology in Natural Language Processing (NLP) to accurately locate core entities such as disease diagnoses and pathological features in the report. The standardization mapping is based on the standard terminology system in the medical field to eliminate noise caused by abbreviations, synonyms, and non-standard descriptions in the clinical report, ensuring the consistency and accuracy of diagnostic labels, and obtaining structured diagnostic label data.

[0071] S2. Obtain raw electrocardiogram (ECG) data. Input the raw ECG data into a multi-granularity ECG encoder for cross-channel slicing and routing aggregation to obtain ECG feature data.

[0072] For example, the ECG analysis terminal acquires raw ECG data from the digital storage module of the ECG acquisition device. This data consists of continuous temporal electrophysiological signals across multiple leads, containing core pathological information such as waveforms and rhythms of cardiac electrical activity, serving as a basis for diagnosing heart diseases. The ECG analysis terminal inputs the raw ECG data into a pre-constructed multi-granularity ECG encoder, which is specifically designed for ECG signal feature extraction and features cross-channel slicing and routing aggregation. Cross-channel slicing treats the multi-lead ECG signal as a whole for multi-scale segmentation, preserving the correlation features between leads and the temporal features at different scales. Routing aggregation uses an attention mechanism to aggregate long-sequence slice features into fixed-length feature tokens, reducing computational complexity while retaining core feature information. The ECG analysis terminal uses this encoder to perform feature extraction and aggregation on the raw ECG data, obtaining ECG feature data to characterize the multi-scale spatiotemporal features of the ECG signal, from local waveform details to global long-range rhythms.

[0073] S3. Input the structured diagnostic label data and ECG feature data into the multimodal fusion network for text semantic extraction and cross-modal interaction to obtain text semantic features and fusion features.

[0074] For example, the ECG analysis terminal inputs structured diagnostic label data into a pre-constructed multimodal fusion network. This network integrates text semantic extraction and cross-modal feature interaction functions, enabling semantic representation of diagnostic label text and cross-modal fusion of ECG-text features. The terminal first extracts text semantics from the structured diagnostic label data using the network's text encoding module, transforming discrete standardized diagnostic labels into continuous high-dimensional text semantic features. These features accurately capture the deep semantic information of the diagnostic labels. The terminal then uses the network's cross-modal fusion module to perform cross-modal interactive calculations on the extracted text semantic features and ECG feature data. Using the text semantic features as constraints, it dynamically filters pathological feature information related to diagnostic semantics from the ECG feature data, achieving complementary enhancement between ECG features and text semantics. The ECG analysis terminal obtains text semantic features representing the semantic information of the diagnostic labels and fused features representing the core pathological features of the ECG signal under text semantic constraints. This fused feature is a cross-modal feature representation, more closely aligned with the semantic logic of clinical diagnosis.

[0075] S4. Based on ECG feature data, text semantic features, and fusion features, modality consistency loss and classification loss are constructed. Based on modality consistency loss and classification loss, the multi-granularity ECG encoder and multi-modal fusion network are jointly optimized to obtain the ECG analysis model.

[0076] Optionally, the ECG analysis terminal uses ECG feature data, text semantic features, and fused features as core inputs. It constructs modality consistency loss and classification loss based on loss calculation methods. These two types of losses constrain the model from two dimensions: cross-modal feature alignment and disease classification performance. Modality consistency loss is constructed based on the similarity calculation between ECG features and text semantic features. It is used to bring semantically matching ECG-text feature pairs closer together and push away semantically mismatched feature pairs in the feature space, achieving fine-grained alignment of cross-modal features. Classification loss is constructed based on the classification prediction results of fused features and the true diagnostic labels. It is used to allow the model to learn the mapping relationship between fused features and cardiac disease diagnostic categories, improving the model's disease classification ability. The ECG analysis terminal weightedly fuses the modality consistency loss and classification loss to obtain an overall loss function. Using this function as the optimization objective, a gradient descent optimization algorithm is employed to backpropagate and update all learnable parameters of the multi-granularity ECG encoder and the multi-modal fusion network. Through multiple rounds of training and parameter optimization, both types of losses converge to stable values, resulting in an electrocardiogram analysis model with cross-modal feature alignment and high-precision disease classification.

[0077] S5. Input the ECG data to be tested into the ECG analysis model for feature encoding and classification prediction to obtain the ECG analysis results.

[0078] For example, the ECG analysis terminal deploys a jointly optimized ECG analysis model. This model integrates feature extraction from a multi-granularity ECG encoder and cross-modal fusion from a multi-modal fusion network, achieving both high-precision feature representation and disease classification performance. When an ECG diagnostic request is received, the terminal acquires the ECG data to be tested from the ECG acquisition device. This data is the undiagnosed multi-lead raw ECG time-series signal, consistent in format and dimension with the raw ECG data used during model training, ensuring effective feature processing by the model. The ECG analysis terminal inputs the ECG data to be tested into the deployed ECG analysis model. The model first performs feature encoding operations such as cross-channel slicing and route aggregation using a multi-granularity ECG encoder to extract multi-scale ECG feature data from the ECG data. The extracted feature data is then input into the multi-modal fusion network for cross-modal feature interaction and classification prediction. Based on the feature-diagnosis category mapping learned during model training, the model outputs the probability of existence of each cardiac disease diagnosis category corresponding to the ECG data. The terminal integrates diagnostic categories with probabilities higher than a preset probability threshold into ECG analysis results. These results are structured disease diagnosis conclusions, which can provide clinicians with objective and accurate auxiliary references for ECG diagnosis.

[0079] The aforementioned supervised pre-training-based multimodal ECG signal representation learning method fully mines and utilizes diagnostic information from clinical text reports. It eliminates inconsistencies in terminology and noise interference in reports through entity extraction and standardized mapping, transforming unstructured text into high-quality supervised signals and overcoming the reliance on large amounts of manually labeled data for supervised learning. Multi-granularity ECG encoding is used to capture multi-scale features of local waveforms and global rhythms in ECG signals. The routing aggregation mechanism significantly reduces the computational complexity of long-term signals. Combined with fine-grained cross-modal interaction in a multimodal fusion network and modal consistency loss, it achieves accurate alignment of ECG and text features, overcoming the shortcomings of single-modal methods that ignore text semantics and the poor performance of traditional multimodal fusion. The jointly optimized model can more fully understand ECG pathological features, significantly improving the accuracy and robustness of the output analysis results, and providing reliable auxiliary support for clinical cardiac disease diagnosis.

[0080] To further illustrate the solutions of the embodiments of this application, the following description uses a specific example applied to the above-mentioned supervised pre-training-based multimodal electrocardiogram signal representation learning method. Figure 2 This is a flowchart illustrating a supervised pre-training-based multimodal ECG signal representation learning method, including the following steps:

[0081] Step 1: Construct a multimodal pre-training dataset. This aims to leverage the powerful semantic understanding capabilities of large language models to extract high-quality supervision signals from unstructured clinical text reports. The specific processing flow is as follows:

[0082] Step 1.1: Construct prompts incorporating knowledge from the cardiology domain. This domain knowledge includes standard ECG terminology (such as the SCP-ECG standard), common clinical abbreviations (such as LVH for left ventricular hypertrophy), and descriptions of waveform abnormalities. Simultaneously, include few sample examples in the prompts to guide the large model to output in a standardized format.

[0083] Step 1.2: Input the prompt instruction and the original ECG report into a large language model (e.g., LLaMA-3-Instruct) for instruction fine-tuning. The model parses the report content and outputs structured text containing diagnostic entities and their deterministic attributes (e.g., "confirmed," "possible," "pending").

[0084] Step 1.3: Filter out entities with a certainty below a preset threshold (e.g., remove uncertain descriptions such as "possible" or "not excluded"). Then, use a pre-trained medical text encoder (such as MedCPT) to map the remaining entities into high-dimensional semantic vectors.

[0085] Step 1.4: Calculate the cosine similarity between the extracted entity vectors and the term vectors in a standard terminology database (such as SNOMED CT). Perform soft matching based on the similarity; for example, a successful match is considered when the similarity is greater than 0.9. Map the entities to standard terms, thus completing deduplication and standardization. Finally, construct a dataset of structured diagnostic label pairs.

[0086] Step 2: Construct a multimodal pre-trained network model, which mainly includes four core components:

[0087] ECG encoder: Used to extract multi-scale spatiotemporal features of ECG signals. Specifically configured as an improved Medformer architecture, it includes cross-channel multi-granularity slicing and routing aggregation units.

[0088] Text encoder: Used to extract semantic features of diagnostic labels, employing a pre-trained language model (such as MedCPT or BERT).

[0089] Multimodal fusion module: used to fuse ECG and text features, employing Transformer Query Network.

[0090] Modality Consistency Learning Module: Used to calculate feature alignment loss.

[0091] Step 3: ECG Feature Encoding (Improved Medformer) The ECG signal is input into the ECG encoder. For example... Figure 3 The diagram shown illustrates the structure of an ECG feature encoder. The encoder's processing involves the following sub-steps, designed to efficiently extract multi-scale features:

[0092] Step 3.1: Preset multiple time windows of different sizes as slice sizes, for example, P={32,64,128}, corresponding to fine-grained, medium-grained, and coarse-grained temporal features, respectively. For each slice size, treat all 12 leads of ECG data within the same time window as a single slice. Use a linear projection layer to map this slice to a feature dimension D, generating multiple slice feature sequences corresponding to different granularities. Simultaneously, add independent learnable positional encodings to each sequence to preserve temporal information.

[0093] Step 3.2: During training, to prevent overfitting, random augmentation is performed on the slice feature sequences after linear projection. Specific operations include: performing random masking with a preset probability (e.g., 0.15) to set some slice features to zero; or adding Gaussian noise that conforms to a normal distribution.

[0094] Step 3.3: Independently input each group of slice feature sequences into a multi-layer Transformer module (e.g., 12 layers), perform self-attention computation, and extract long-range temporal dependency features at each granularity. At this stage, multi-level feature fusion is preferred: during the Transformer computation, extract the network outputs from the shallow (e.g., layer 1), middle (e.g., layer 6), and deep (e.g., layer 12) layers respectively. Concatenate or add these three layers of features along the channel dimension, and map them back to the original dimension through a projection layer. This step ensures that the feature sequence simultaneously contains both low-level waveform morphology information and high-level semantic information.

[0095] Step 3.4: Initialize an independent learnable vector, called a routing token, for each granularity. Utilizing a cross-attention mechanism: use the routing token as the query, and the variable-length slice feature sequence output from the third stage of that granularity as the key and value; through attention calculation, key information from the long sequence is dynamically aggregated into the fixed-length routing token. Regardless of the length of the original ECG signal, after this step, each granularity outputs only a representative feature vector.

[0096] Step 3.5: Stack all the aggregated routing tokens of all granularities (e.g., 3 tokens for 3 granularities) and perform self-attention computation. This allows information from different time scales (such as fine-grained and coarse-grained) to interact and fuse, outputting the final enhanced ECG feature representation.

[0097] Step 4: Text encoding and multimodal fusion, including the following processes.

[0098] Step 4.1: Input the structured diagnostic labels generated in Step 1 into the text encoder to extract the text semantic features.

[0099] Step 4.2: Interact through the multimodal fusion module. This embodiment uses a Transformer query network: text semantic features are used as input queries for the decoder, and ECG feature representations (i.e., routing token sequences) are used as encoders for the decoder. Using a multi-layer cross-attention mechanism, the text vector "queries" relevant pathological patterns in the ECG features. Information is dynamically aggregated according to attention weights to generate fusion features for classification and output classification prediction results (Logits).

[0100] Step 5: To further enhance the alignment between ECG features and text features in the latent space, a modality consistency learning module is introduced. For example... Figure 4 The diagram shown illustrates the modal consistency loss module. The specific calculation process is as follows:

[0101] Step 5.1: Perform L2 normalization on the ECG feature representation and text semantic features respectively to ensure that the feature distribution is on a unit sphere.

[0102] Step 5.2: Calculate the cosine similarity matrix S between the normalized ECG feature matrix and the text feature matrix.

[0103] Step 5.3: Construct the structured diagnostic labels into multi-label binary classification vectors. Based on this, generate a positive sample mask matrix (positions with label 1) and a negative sample mask matrix (positions with label 0) with the same dimensions as the similarity matrix.

[0104] Step 5.4, Consistency Loss Calculation:

[0105] Positive sample loss: Based on the positive sample mask, calculate the difference between the similarity at the corresponding position and 1 (1−Spos), aiming to shorten the distance between matching pairs.

[0106] Negative sample loss: Based on a negative sample mask, calculate the difference between the similarity at the corresponding position and a preset boundary value (Margin, e.g., 0.2). Using the ReLU function or truncation operation, the loss (max0, Sneg−Margin) is calculated only when the similarity is greater than the Margin, aiming to push away mismatched pairs.

[0107] Step 5.5: Calculate the mean of all positive sample losses and negative sample losses to obtain the final consistency loss. Weight this loss and sum it with the classification task loss (such as BCE Loss), and then use the backpropagation algorithm to perform end-to-end optimization of the multimodal pre-trained network.

[0108] To further illustrate the solutions of the embodiments of this application, the following description uses a specific example applied to the above-mentioned supervised pre-training-based multimodal electrocardiogram signal representation learning method. Figure 5A schematic diagram of a multimodal electrocardiogram signal representation learning system 10 is provided. The system includes:

[0109] Data processing module 11: Used to call the large language model to process the raw ECG report, perform prompting engineering, entity extraction, semantic calculation and standardization mapping, and build a structured diagnostic label set.

[0110] ECG Feature Extraction Module 12: Equipped with a Medformer variant encoder. This module internally includes a cross-channel slicing unit, a data augmentation unit, an intra-granularity attention unit, a route aggregation unit (performing cross-attention), and an inter-granularity interaction unit.

[0111] Text feature extraction module 13: It is equipped with a pre-trained language model to transform discrete labels into continuous semantic vectors.

[0112] Multimodal fusion module 14: Configured with a Transformer query network. This module receives textual semantic features as a decoder query, receives ECG feature representations as an encoder storage, performs feature interaction and fusion through a cross-attention mechanism, and outputs classification prediction results.

[0113] Training Optimization Module 15: Configures the construction of positive and negative sample masks, calculates the classification loss based on the classification prediction results, calculates the modality consistency loss based on feature similarity, and updates all learnable parameters using the gradient descent algorithm.

[0114] In an optional embodiment, S1 includes:

[0115] S11. Based on the standard ECG terminology database and common clinical abbreviations, construct prompts containing knowledge of the cardiology field. Concatenate the prompts and clinical text reports and input them into the large language model for fine-tuning of the instructions to parse the clinical text reports and output structured text containing diagnostic entity names and deterministic values.

[0116] Optionally, the ECG analysis terminal first constructs prompts containing cardiology knowledge based on an industry-standard ECG terminology database and frequently occurring ECG-related abbreviations in clinical practice. These prompts, in addition to incorporating the mapping relationship between standard terms and abbreviations, also include a small number of diagnostic report parsing examples to guide the model to output results in a fixed format. The ECG analysis terminal then concatenates the constructed prompts with the clinical text report to generate a text sequence conforming to the input specifications of the Large Language Model (LLM). This LLM is then input into the LLM after fine-tuning with medical corpora, resulting in enhanced semantic understanding and entity parsing capabilities for medical texts. The model performs semantic parsing and entity extraction on the input text sequence, accurately identifying diagnostic entity names in the report. Based on its semantic understanding capabilities, it assigns a corresponding deterministic value to each diagnostic entity. This value represents the model's confidence in the match between the extracted diagnostic entity and the actual diagnostic result in the report. Finally, it outputs structured text containing diagnostic entity names and deterministic values, formatted as preset key-value pairs.

[0117] S12. Filter out diagnostic entities in the structured text whose deterministic values ​​are lower than a preset threshold to obtain a set of high-confidence diagnostic entities.

[0118] Optionally, after the ECG analysis terminal obtains the structured text containing diagnostic entity names and deterministic values ​​from the output of the large language model, it first performs structured parsing on the data in the text to extract the deterministic value corresponding to each diagnostic entity. This value is a quantitative result of the model's confidence in the diagnostic entity, and the value reflects the reliability of the diagnostic entity. The ECG analysis terminal compares the parsed deterministic values ​​one by one with a preset threshold. This preset threshold is a confidence threshold set based on the reliability requirements of medical diagnosis, used to eliminate low-confidence diagnostic entities generated during the model recognition process, avoiding interference from noisy entities in subsequent standardized mapping. The ECG analysis terminal filters out all diagnostic entities with deterministic values ​​below the preset threshold, retaining only those with deterministic values ​​that meet the requirements. These high-reliability diagnostic entities are integrated into a high-confidence diagnostic entity set, which is a noise-free, high-confidence diagnostic entity dataset.

[0119] S13. Input each diagnostic entity in the high-confidence diagnostic entity set into the pre-trained medical text encoder for semantic encoding to obtain entity semantic vectors. Input each standard term in the standard ECG terminology library into the medical text encoder for semantic encoding to obtain a standard semantic vector library.

[0120] Optionally, the ECG analysis terminal takes each diagnostic entity in the high-confidence diagnostic entity set as an independent input and sequentially inputs it into a pre-trained medical text encoder. This encoder is a language model pre-trained on a massive medical text corpus, capable of converting medical text entities into high-dimensional semantic vectors and capturing the deep semantic features of diagnostic entities. The ECG analysis terminal performs semantic encoding processing on each diagnostic entity through this encoder, transforming discrete text entities into continuous high-dimensional entity semantic vectors, thus achieving a numerical representation of the diagnostic entities. The ECG analysis terminal inputs all standard terms from the standard ECG terminology library one by one into the medical text encoder, performing the same semantic encoding operation to convert each standard term into a corresponding high-dimensional standard semantic vector. The ECG analysis terminal integrates all standard semantic vectors into a standard semantic vector library. This vector library provides a reference benchmark for the standardized mapping of diagnostic entities. The standard semantic vectors it contains are all semantic representations of standardized medical terms, ensuring the standardization and uniformity of subsequent mapping results.

[0121] S14. Calculate the cosine similarity between the semantic vector of each entity and all vectors in the standard semantic vector library. Select the standard term corresponding to the maximum cosine similarity as the standardized mapping result of the diagnostic entity. Keep the diagnostic entities with cosine similarity lower than the preset matching threshold as new temporary terms. Merge and deduplicate the diagnostic entities to generate structured diagnostic label data.

[0122] Optionally, the ECG analysis terminal calculates the cosine similarity between the semantic vector of each entity in the high-confidence diagnostic entity set and all standard semantic vectors in the standard semantic vector library, based on the cosine similarity calculation method. This similarity quantifies the semantic matching degree between the diagnostic entity and the standard terminology; a higher value indicates a stronger semantic association. The ECG analysis terminal selects the standard term corresponding to the maximum cosine similarity in the calculation results for each entity's semantic vector and uses it as the standardized mapping result for that diagnostic entity, realizing the conversion of non-standard diagnostic entities into standard terms. For diagnostic entities with a cosine similarity lower than a preset matching threshold, the ECG analysis terminal retains them as newly added temporary terms, compatible with emerging diagnostic expressions or special pathological descriptions not yet included in the standard terminology library in clinical practice. The ECG analysis terminal performs a merging and deduplication operation on the mapped and retained diagnostic entities, removing synonymous and duplicate entities from the set, generating structured diagnostic label data composed of standardized terms and compliant temporary terms, ensuring the uniqueness and standardization of the data.

[0123] In an optional embodiment, S2 includes:

[0124] S21. Preset at least two sets of time windows of different sizes as slice granularities. For each slice granularity, construct a three-dimensional tensor slice from all leads of ECG data points within the same time window. Map the three-dimensional tensor slice to a one-dimensional feature vector through a learnable linear projection layer to obtain the slice feature sequence at each granularity. Add independent learnable positional codes to the slice feature sequences at each granularity.

[0125] Optionally, the ECG analysis terminal pre-sets at least two sets of time windows of different sizes as slice granularities. Different sized time windows correspond to different scale features of the ECG signal; smaller windows capture local waveform details, while larger windows capture global rhythm patterns. For each preset slice granularity, the ECG analysis terminal integrates all ECG data points from all leads within the same time window in the original ECG data into a three-dimensional tensor slice. This tensor slice retains the temporal features of the time dimension and the channel correlation features of the lead dimension, avoiding the loss of inter-lead information caused by single-lead slices. The ECG analysis terminal configures an independent learnable linear projection layer for each slice granularity, linearly transforming the three-dimensional tensor slice through this projection layer to map it into a fixed-dimensional one-dimensional feature vector. All one-dimensional feature vectors under the same granularity are arranged in chronological order to obtain the slice feature sequence for the corresponding granularity. The ECG analysis terminal adds independent learnable position codes to the slice feature sequences of each granularity to represent the temporal position information of the slice in the original ECG signal, compensating for the temporal position features lost during linear projection.

[0126] S22. Input the slice feature sequence at each granularity into the multi-layer Transformer module for self-attention calculation. During the calculation process of the multi-layer Transformer module, extract the first output feature of the shallow network layer, the second output feature of the middle network layer, and the third output feature of the deep network layer. After concatenating the first output feature, the second output feature, and the third output feature, perform dimensionality reduction mapping through the first fully connected projection layer to obtain the enhanced slice feature sequence.

[0127] Optionally, the ECG analysis terminal inputs the slice feature sequences at each granularity into independently configured multi-layer Transformer modules. These modules, composed of multiple self-attention layers and feedforward network layers, are capable of capturing long-range temporal dependencies in sequence data. The terminal performs self-attention computation on the slice feature sequences through the multi-layer Transformer modules. During the computation, it extracts the first output feature from the shallow network layer, the second output feature from the middle network layer, and the third output feature from the deep network layer according to the distribution of the network layers. The first output feature from the shallow network layer represents the underlying waveform morphology information of the ECG signal, the second output feature from the middle network layer represents local temporal correlation information, and the third output feature from the deep network layer represents high-level pathological semantic information. The ECG analysis terminal concatenates the extracted first, second, and third output features along the feature dimension, integrating the feature information from multiple network layers to obtain a fused feature sequence. The terminal inputs the fused feature sequence into a first fully connected projection layer for dimensionality reduction mapping, mapping the high-dimensional fused feature sequence into a fixed-dimensional feature sequence, resulting in an enhanced slice feature sequence.

[0128] S23. Initialize a learnable routing token for each granularity. The learnable routing token is a trainable parameter vector with the same feature dimension as the slice feature sequence. For each granularity, use the learnable routing token as the query matrix and the enhanced slice feature sequence under the granularity as the key matrix and value matrix. Calculate the attention weights through a multi-head cross-attention mechanism, and aggregate the enhanced slice feature sequence into the learnable routing token according to the attention weights to output the routing token of the granularity.

[0129] Optionally, the ECG analysis terminal initializes a learnable routing token for each slice granularity. This routing token is a trainable parameter vector with the same feature dimensions as the slice feature sequence, serving as the aggregation core for the corresponding granularity's slice features. Its parameters are continuously optimized during model training. For each slice granularity, the ECG analysis terminal uses the learnable routing token corresponding to that granularity as a query matrix, and the processed enhanced slice feature sequence as the key and value matrices, respectively, inputting them into a multi-head cross-attention mechanism for computation. This mechanism uses multiple attention heads to calculate the attention weights between the routing token and each feature vector in the enhanced slice feature sequence in parallel. The weight magnitude is used to characterize the importance of the feature vector to the granularity feature. The ECG analysis terminal performs a weighted summation of the enhanced slice feature sequence according to the calculated attention weights, and aggregates the weighted feature information into the learnable routing token, completing the compression of long sequence features into a fixed-length token. It outputs the aggregated routing token corresponding to each granularity, which is the core representation of the ECG feature at the corresponding granularity, reducing the computational complexity of subsequent feature processing.

[0130] S24. Stack the routing tokens corresponding to at least two granularities in the sequence dimension to generate a token sequence. Input the token sequence into the Transformer encoder layer for self-attention calculation. After the self-attention calculation, concatenate all the routing tokens of all granularities output in the feature dimension and then fuse and transform them through the second fully connected projection layer to obtain ECG feature data.

[0131] Optionally, the ECG analysis terminal stacks the aggregated routing tokens corresponding to at least two slice granularities in the sequence dimension, combining them into a one-dimensional token sequence according to the size order of the slice granularity. This sequence integrates the core feature information of ECG signals at different scales. The ECG analysis terminal inputs the generated token sequence into a pre-configured Transformer encoder layer, which performs self-attention calculation on the token sequence to achieve information interaction and feature fusion between routing tokens of different granularities, allowing the features of each granularity to complement each other and overcome the limitations of single-granularity feature representation. After the calculation is completed, the ECG analysis terminal concatenates all granularity routing tokens output by the Transformer encoder layer in the feature dimension to obtain a high-dimensional fused token vector integrating multi-scale features. The terminal inputs this fused token vector into a second fully connected projection layer for feature fusion and dimensional transformation operations, mapping the high-dimensional fused token vector into a fixed-dimensional high-dimensional feature vector to obtain ECG feature data, which is used to represent the spatiotemporal features of ECG signals at different scales.

[0132] In an optional embodiment, S3 includes:

[0133] S31. Perform word segmentation on the diagnostic label text in the structured diagnostic label data to obtain a word index sequence, and convert the word index sequence into a word vector sequence through a word embedding layer.

[0134] Optionally, the ECG analysis terminal performs word segmentation on the diagnostic label text in the structured diagnostic label data based on a medical-specific word segmentation dictionary, identifying medical terms within the diagnostic labels. The terminal breaks down the continuous diagnostic label text into independent lexical units through word segmentation, then maps each lexical unit to a pre-defined unique integer index. All integer indices are arranged according to the semantic order of the text, resulting in a word index sequence, thus transforming the diagnostic label text from character form to an integer sequence. The ECG analysis terminal inputs the generated word index sequence into a learnable word embedding layer. This learnable word embedding layer is the foundational layer of the neural network, containing a medical-specific lexical embedding matrix. It maps each integer index to a fixed-dimensional high-dimensional word vector. All word vectors are arranged according to the word index sequence, resulting in a word vector sequence, achieving both numerical and vectorized representation of the diagnostic label text. The word vector sequence is used to initially capture the semantic features of the words.

[0135] S32. Input the word vector sequence into the Transformer encoder of the pre-trained language model to perform context modeling, obtain the hidden state vector corresponding to each word, and extract the output vector corresponding to the special marker as the semantic representation of the entire diagnostic label text.

[0136] Optionally, the ECG analysis terminal inputs the word vector sequence into a pre-trained language model's Transformer encoder. This encoder, pre-trained on a massive medical text corpus, possesses powerful contextual semantic modeling capabilities, capturing the semantic relationships and overall semantic logic between words in the diagnostic label text. The terminal uses this Transformer encoder to perform multi-layer self-attention computation and feedforward network transformation on the word vector sequence. During the computation, the word vector of each word is updated in conjunction with information from other words in the context, ultimately obtaining the hidden state vector corresponding to each word. This vector is a high-dimensional feature vector that integrates contextual semantic information, used to represent the semantic meaning of the word in the specific diagnostic label context. At the beginning of the word vector sequence in the diagnostic label text, the ECG analysis terminal pre-adds a special marker. This marker has no actual semantic meaning but is designed specifically for extracting the overall semantic representation of the text. The terminal extracts the output vector of this special marker at the last layer of the Transformer encoder and uses it as the semantic representation of the entire diagnostic label text. This vector integrates all the contextual semantic information of the diagnostic label text, providing a holistic semantic representation of the text.

[0137] S33. Perform dimensional transformation on the semantic representation to obtain the text semantic features.

[0138] Optionally, after acquiring the diagnostic label text semantic representation output by the Transformer encoder, the ECG analysis terminal first analyzes the feature dimensions of this semantic representation. Since its dimension matches the output dimension of the pre-trained language model, it may not match the feature dimensions for subsequent cross-modal interaction. To achieve cross-modal interaction with ECG feature data, the ECG analysis terminal inputs this semantic representation into a pre-configured linear transformation layer. This layer consists of a learnable weight matrix and a bias vector. Through linear transformation operations, the feature dimensions of the semantic representation are adjusted, mapping it to a high-dimensional feature vector that matches the feature dimensions of the ECG feature data. During the dimensionality transformation process, this linear transformation layer only scales and reorganizes the feature dimensions, avoiding the loss of semantic information. The ECG analysis terminal obtains text semantic features with adapted dimensions and complete semantic information.

[0139] In an optional embodiment, S3 further includes:

[0140] S41. Input the ECG feature data into the linear extension layer for transformation to obtain the ECG feature sequence, and use the ECG feature sequence as the encoder memory in the Transformer decoder architecture.

[0141] Optionally, the ECG analysis terminal inputs the extracted ECG feature data into a pre-configured linear extension layer. This layer consists of learnable linear transformation matrices used to adjust the feature dimension and sequence length of the ECG feature data. The terminal performs a linear transformation operation on the ECG feature data through this linear extension layer, mapping the single-vector form of the ECG feature data into a fixed-length sequence-type feature vector, resulting in an ECG feature sequence. The feature dimension of this sequence matches the input dimension of the subsequent Transformer decoder, and the sequence length is set according to the multi-scale information of the ECG features to ensure complete representation of the multi-scale features of the ECG signal. The ECG analysis terminal uses the generated ECG feature sequence as the encoder memory in the Transformer decoder architecture. This encoder memory provides the basic data for the key and value matrices for the Transformer decoder's cross-modal attention calculation. It is called by each layer of the decoder and is the core signal feature data source for realizing cross-modal interaction between text semantics and ECG features, ensuring that the decoder can dynamically filter feature information related to text semantics from the ECG feature sequence.

[0142] S42. Using text semantic features as the initial input query vector of the decoder in the Transformer decoder architecture, in each layer of the Transformer decoder, the decoder input query vector is used as the query matrix, and the ECG feature sequence is used as the key matrix and value matrix. The interaction weight between text semantic features and ECG feature sequence is calculated. Based on the interaction weight, the text semantic features are dynamically aggregated from the ECG feature sequence to generate the decoder output of the current layer.

[0143] Optionally, the ECG analysis terminal uses the obtained text semantic features as the initial input query vector of the decoder in the Transformer decoder architecture. This vector serves as the query benchmark for the decoder's cross-modal attention calculation, and the text semantic information it carries guides the decoder to select relevant features from the ECG feature sequence. In each layer of the Transformer decoder network, the ECG analysis terminal uses the current layer's decoder input query vector as the query matrix and the pre-set encoder memory, i.e., the ECG feature sequence, as the key and value matrices to perform cross-modal attention calculation. This calculation obtains the interaction weights between the text semantic features and the ECG feature sequence by solving the similarity between the query vector and each feature vector in the ECG feature sequence. The weight magnitude is used to characterize the degree of correlation between the corresponding feature in the ECG feature sequence and the text semantics. Based on the calculated interaction weights, the terminal performs weighted aggregation on the feature vectors in the ECG feature sequence, fusing ECG feature information highly correlated with the text semantic features into the decoder input query vector to generate the current layer's decoder output. This output is a vector that fuses the cross-modal feature information of the corresponding layer, serving as the input query vector for the next layer's decoder.

[0144] S43. The feature vector output by the last decoder layer is used as the fusion feature, and the fusion feature is input into the classification head composed of fully connected layers to obtain the classification prediction result.

[0145] Optionally, the ECG analysis terminal allows each layer of the Transformer decoder to sequentially perform cross-modal attention calculation and feature fusion operations. The decoder output of each layer serves as the input query vector for the next layer, continuously fusing relevant information from the ECG feature sequence to obtain the feature vector output by the final decoder layer. This feature vector fully integrates the semantic constraints of the textual semantic features and the core pathological features of the ECG feature sequence. The ECG analysis terminal uses this as a fused feature, which is a cross-modal feature representation used to reflect the core pathological features of the ECG signal under the corresponding diagnostic semantic constraints. The ECG analysis terminal inputs the fused feature into a classification head composed of fully connected layers. This classification head consists of at least one stacked fully connected layer, and its output dimension is consistent with the preset number of cardiac disease diagnostic categories. The classification head performs feature transformation and dimension mapping on the fused feature through fully connected layers, mapping the high-dimensional fused feature into a feature vector matching the number of diagnostic categories, and finally obtains the classification prediction result, which is the preliminary predicted score for each diagnostic category.

[0146] In an optional embodiment, S4 includes:

[0147] S51. Perform L2 norm normalization on the ECG feature data and text semantic features respectively to obtain normalized ECG features and normalized text features.

[0148] Optionally, to ensure fair similarity calculation between ECG feature data and text semantic features, the ECG analysis terminal first performs L2 norm normalization on both the ECG feature data and the text semantic features. This normalization process scales the feature vector's magnitude to 1 by solving for the L2 norm of the feature vector, ensuring that all feature vectors are distributed on a unit hypersphere. This eliminates similarity calculation bias caused by differences in magnitude between different feature vectors, ensuring that the similarity is determined solely by the angle between the feature vectors, thus accurately reflecting the semantic correlation between ECG features and text semantic features. During the normalization process, the ECG analysis terminal normalizes the feature values ​​of each dimension of the ECG feature data to obtain normalized ECG features with unchanged dimensions and a magnitude of 1. Simultaneously, the same L2 norm normalization operation is performed on the text semantic features to obtain normalized text features. The two types of normalized feature vectors retain the directional information of the original features, i.e., the core feature representation information, with only the magnitude being standardized, ensuring the objectivity and accuracy of the calculation results.

[0149] S52. In a training batch, calculate the cosine similarity between all normalized ECG features and all normalized text features to obtain the cosine similarity matrix. The expression for cosine similarity is:

[0150]

[0151] in, Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of text features of each sample. Indicates the first in the batch ECG characteristic data of each sample Indicates the first in the batch Textual semantic features of each sample This represents L2 norm operations.

[0152] Specifically, in a training batch of model training, the ECG analysis terminal uses the normalized ECG features and normalized text features of all samples within that batch as calculation objects. Based on the aforementioned cosine similarity expression, it calculates the cosine similarity between the normalized ECG features of any sample and the normalized text features of any sample, one by one. In the aforementioned cosine similarity expression, Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of text features of each sample. Indicates the first in the batch ECG characteristic data of each sample Indicates the first in the batch Textual semantic features of each sample The L2 norm operation is calculated by taking the dot product of two normalized feature vectors. Since the feature vectors have been normalized using the L2 norm, the dot product result is the cosine similarity between the two vectors. The value ranges from -1 to 1; a larger value indicates that the feature directions are closer and the semantic association is stronger. The ECG analysis terminal arranges all the calculated cosine similarities according to the batch order of the samples to construct a cosine similarity matrix, which is used to characterize the semantic association of all ECG-text feature pairs within a batch.

[0153] S53. Construct a multi-label vector for each sample within a batch based on the structured diagnostic label data. Calculate the label similarity between any two sample multi-label vectors to obtain a label similarity matrix. Set a positive sample threshold and a negative sample threshold. Map the positions in the label similarity matrix that are greater than the positive sample threshold to the same positions in the cosine similarity matrix and assign them a value of 1 to construct a positive sample mask matrix. Map the positions in the label similarity matrix that are less than the negative sample threshold to the same positions in the cosine similarity matrix and assign them a value of 1 to construct a negative sample mask matrix.

[0154] Optionally, the ECG analysis terminal constructs a corresponding multi-label vector for each sample in the training batch based on structured diagnostic label data. The dimension of this vector is consistent with the preset number of cardiac disease diagnostic categories. Each element in the vector is labeled as 0 or 1 according to the diagnostic label, indicating whether the sample contains the diagnostic result of the corresponding category, respectively. The terminal calculates the similarity between the multi-label vectors of any two samples to obtain a label similarity matrix, which is used to characterize the degree of matching of diagnostic categories between samples in the batch. The terminal pre-sets positive sample thresholds and negative sample thresholds, with the positive sample threshold being the high similarity threshold and the negative sample threshold being the low similarity threshold. For positions in the label similarity matrix where the value is greater than the positive sample threshold, the terminal assigns a value of 1 in the same row and column position of the cosine similarity matrix, and assigns a value of 0 in the remaining positions, constructing a positive sample mask matrix, which is used to label semantically matched positive sample ECG-text feature pairs. For positions in the label similarity matrix where the value is less than the negative sample threshold, the terminal assigns a value of 1 to the same row and column position in the cosine similarity matrix, and assigns a value of 0 to the other positions, thus constructing a negative sample mask matrix, which is used to mark semantically mismatched negative ECG-text feature pairs.

[0155] S54. Based on the positive sample mask matrix, the positive sample loss is calculated. The expression for the positive sample loss is:

[0156]

[0157] in, Indicates positive sample loss. Indicates batch size, Represents the positive sample mask matrix. Line 1 Column elements, This represents the sum of all elements in the positive sample mask matrix. Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of text features among samples.

[0158] Optionally, the ECG analysis terminal calculates the positive sample loss based on the constructed positive sample mask matrix. The optimization objective of this loss is to make the cosine similarity of semantically matched positive sample ECG-text feature pairs as close to 1 as possible, achieving close alignment of cross-modal features. In the above expression for the positive sample loss, Indicates positive sample loss. Indicates batch size, Represents the positive sample mask matrix. Line 1 Column elements, This represents the sum of all elements in the positive sample mask matrix. Indicates the first in the batch The electrocardiogram characteristics of the sample and the first For the similarity of text features of each sample, the terminal, based on the aforementioned expression for positive sample loss, first filters the cosine similarity matrix using the positive sample mask matrix, retaining only the cosine similarity corresponding to the position where the element in the mask matrix is ​​1, and discarding irrelevant data at other positions. The terminal calculates the difference between each cosine similarity after filtering and 1. This difference represents the deviation of the similarity of the positive sample feature pair from the ideal value; the larger the difference, the worse the feature alignment effect. The terminal sums all differences and then divides them by the sum of all elements in the positive sample mask matrix, i.e., the number of positive samples, to obtain the mean value as the positive sample loss. This loss value is non-negative; the smaller the loss value, the better the feature alignment effect of the positive sample ECG-text feature pair. It will serve as an important component of modality consistency loss, guiding the model to narrow the distance between positive sample feature pairs during training.

[0159] S55. Based on the negative sample mask matrix, the negative sample loss is calculated. The expression for the negative sample loss is:

[0160]

[0161] in, Indicates negative sample loss. Indicates batch size, Represents the positive sample mask matrix. Line 1 Column elements, This represents the sum of all elements in the positive sample mask matrix. Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of text features of each sample. This represents the preset boundary value hyperparameters.

[0162] Optionally, the ECG analysis terminal calculates the negative sample loss based on the constructed negative sample mask matrix. The optimization objective of this loss is to ensure that the cosine similarity of semantically mismatched negative sample ECG-text feature pairs is as low as possible below a preset boundary value, thereby achieving effective separation of cross-modal features. In the above expression for the negative sample loss, Indicates negative sample loss. Indicates batch size, Represents the positive sample mask matrix. Line 1 Column elements, This represents the sum of all elements in the positive sample mask matrix. Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of text features of each sample. This represents the preset boundary value hyperparameter. Based on the aforementioned expression for negative sample loss, the terminal first filters the cosine similarity matrix using a negative sample mask matrix, retaining only the cosine similarity corresponding to the position where the element in the mask matrix is ​​1. The terminal calculates the difference between each filtered cosine similarity and the preset boundary value, truncating the portion of the difference less than 0 to 0 using a maximum value function, retaining only the portion of the difference greater than 0. This portion represents the deviation of the similarity of the negative sample feature pair from the preset boundary value; the larger the deviation, the worse the feature separation effect. The terminal sums all the truncated differences and divides them by the sum of all elements in the negative sample mask matrix, i.e., the number of negative samples, to obtain the mean as the negative sample loss. This loss value is non-negative; the smaller the loss value, the better the feature separation effect of the negative sample ECG-text feature pair, which is another important component of modality consistency loss.

[0163] S56. The modality consistency loss is obtained by weighted summing of the positive sample loss and the negative sample loss. The expression for the modality consistency loss is:

[0164]

[0165] in, This represents the modal consistency loss. Indicates positive sample loss. Indicates negative sample loss. These are the weighting coefficients for the positive sample loss. The weighting coefficients are for the negative sample loss.

[0166] Optionally, to integrate the optimization constraints of positive and negative sample losses, the ECG analysis terminal performs a weighted summation of the calculated positive and negative sample losses to obtain the modal consistency loss. This loss serves as the overall optimization objective for cross-modal feature alignment. The integrated constraint model achieves tight alignment of positive sample feature pairs and effective separation of negative sample feature pairs. Specifically, during the weighted summation process, the terminal assigns weight coefficients to the positive sample loss. Configure weighting coefficients for negative sample loss. The two weight coefficients are adjustable hyperparameters. Their weight ratios can be adjusted according to the actual needs of model training to achieve different optimization focuses on feature alignment and feature separation. For example, they can be increased in the early stages of training. The weights are prioritized to align positive sample feature pairs. The modality consistency loss is determined by the positive sample loss, negative sample loss, and their corresponding weight coefficients. The smaller the loss value, the better the cross-modal feature alignment effect of the model. This loss, together with the classification loss, serves as the objective function for joint optimization of the model, guiding the model to improve the representation and alignment ability of cross-modal features while learning classification capabilities.

[0167] In an optional embodiment, S4 further includes:

[0168] S61. Based on the fusion features, the existence probability of each preset diagnostic category is obtained through the fully connected layer of the classification head and the sigmoid activation function.

[0169] Optionally, the ECG analysis terminal inputs the fused features obtained from cross-modal fusion into the fully connected layer of the classification head. This fully connected layer is the core feature transformation module of the classification head, and its learnable weight matrix maps the high-dimensional fused features into a feature vector consistent with the number of preset cardiac disease diagnostic categories. Each element of this vector corresponds to a predicted score for a diagnostic category. To convert the predicted scores into numerical values ​​that conform to the definition of probability, the ECG analysis terminal inputs the feature vector output from the fully connected layer into a sigmoid activation function. This function can map predicted scores in any real number field to the interval between 0 and 1, realizing the probabilistic transformation of the scores. After processing by the sigmoid activation function, each element in the feature vector becomes a value between 0 and 1. This value represents the probability of a sample belonging to the corresponding preset diagnostic category. The closer the probability value is to 1, the greater the likelihood that the sample contains that diagnostic category; the closer it is to 0, the smaller the likelihood. The terminal integrates the existence probabilities of all diagnostic categories into a probability vector, which serves as the classification prediction result corresponding to the fused features.

[0170] S62. Construct a multi-label true value vector based on the structured diagnostic label data.

[0171] Optionally, the ECG analysis terminal uses structured diagnostic label data to construct a corresponding multi-label ground truth vector for each sample in the training batch. This vector serves as the ground truth benchmark for calculating the model's classification loss, and its dimension is completely consistent with the preset number of cardiac disease diagnostic categories, ensuring matching with the probability vector dimension of the classification prediction results. For each sample, the terminal assigns a value of 1 to the element corresponding to the diagnostic category in the multi-label ground truth vector based on the diagnostic category included in its structured diagnostic label data, indicating that the sample indeed contains that diagnostic category; for diagnostic categories not included in the structured diagnostic label data, the corresponding element in the vector is assigned a value of 0, indicating that the sample does not contain that diagnostic category. The multi-label ground truth vector is a binary vector containing only 0 and 1 elements, used to represent the actual cardiac disease diagnosis result of the sample. Its construction process strictly follows the annotation information of the structured diagnostic label data to ensure the accuracy and objectivity of the ground truth.

[0172] S63. Calculate the binary cross-entropy loss as the classification loss. The expression for the classification loss is:

[0173]

[0174] in, Represents classification loss, Represents a multi-labeled true value vector. Indicates the first In the nth sample The probability of the existence of each diagnostic category For batch size, This represents the total number of preset diagnostic categories.

[0175] Optionally, the ECG analysis terminal uses binary cross-entropy loss as the classification loss. This loss function is suitable for multi-label classification tasks and can accurately quantify the deviation between the predicted probability and the true value of each diagnostic category. Based on the calculation formula for the classification loss, the terminal calculates the cross-entropy value between the existence probability of each preset diagnostic category and the corresponding element in the multi-label true value vector for each sample in the training batch. The cross-entropy value amplifies the deviation between the predicted probability and the true value through logarithmic operation, achieving strong constraints on model prediction errors. The terminal sums the cross-entropy values ​​of all diagnostic categories for each sample and then takes the average of the summations for all samples in the batch to obtain the final classification loss. This loss value is non-negative; the smaller the loss value, the smaller the deviation between the model's classification prediction result and the actual diagnosis result, and the better the classification performance. The classification loss is weighted and fused with the modality consistency loss as the overall objective function for joint model optimization, guiding the model to improve the multi-label classification accuracy of cardiac diseases while achieving cross-modal feature alignment.

[0176] In the aforementioned supervised pre-training-based multimodal ECG signal representation learning method, entity extraction and standardized mapping of clinical text are achieved through a large language model combined with domain knowledge, transforming unstructured reports into high-quality structured diagnostic labels and fully exploring the diagnostic value of text. The cross-channel slicing design of the multi-granularity ECG encoder can capture multi-scale spatiotemporal features of ECG signals, and the routing aggregation mechanism significantly reduces the computational cost of long sequences. The multimodal fusion network realizes fine-grained cross-modal interaction between text and ECG features, and the modality consistency loss further enhances cross-modal feature alignment, allowing the model to fully integrate text semantics and ECG signal features, improve the understanding of ECG pathological features, enhance the accuracy and robustness of ECG analysis model diagnosis, and improve the model's generalization ability.

[0177] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0178] Based on the same inventive concept, this application also provides a supervised pre-training-based multimodal electrocardiogram (ECG) signal representation learning system for implementing the aforementioned supervised pre-training-based multimodal ECG signal representation learning method. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the supervised pre-training-based multimodal ECG signal representation learning system provided below can be found in the limitations of the supervised pre-training-based multimodal ECG signal representation learning method described above, and will not be repeated here.

[0179] In one exemplary embodiment, such as Figure 6 As shown, a schematic diagram of a supervised pre-training-based multimodal electrocardiogram signal representation learning system 20 is provided, including:

[0180] The text data acquisition and standardization module 21 is used to acquire clinical text reports, extract entities from the clinical text reports and perform standardized mapping to obtain structured diagnostic label data; among which, the structured diagnostic label data is used to characterize the diagnostic categories of heart diseases;

[0181] The ECG data processing module 22 is used to acquire raw ECG data, input the raw ECG data into a multi-granularity ECG encoder for cross-channel slicing and routing aggregation, and obtain ECG feature data; wherein, the ECG feature data is used to characterize the spatiotemporal features of ECG signals at multiple scales;

[0182] The data fusion analysis module 23 is used to input structured diagnostic label data and electrocardiogram feature data into a multimodal fusion network for text semantic extraction and cross-modal interaction to obtain text semantic features and fusion features; among them, the fusion features are used to characterize the pathological features of electrocardiogram signals under text semantic constraints.

[0183] The ECG analysis model training module 24 is used to construct modality consistency loss and classification loss based on ECG feature data, text semantic features and fusion features, and to jointly optimize the multi-granularity ECG encoder and multi-modal fusion network based on the modality consistency loss and classification loss to obtain the ECG analysis model;

[0184] The ECG data analysis module 25 is used to input the ECG data to be tested into the ECG analysis model for feature encoding and classification prediction to obtain the ECG analysis results.

[0185] Furthermore, the text data acquisition and standardization module 21 is also used for:

[0186] S11. Construct prompts containing knowledge of the cardiology field based on the standard ECG terminology database and common clinical abbreviations. Then, concatenate the prompts and clinical text reports and input them into the large language model for fine-tuning of the prompts to parse the clinical text reports and output structured text containing diagnostic entity names and deterministic values.

[0187] S12. Filter out diagnostic entities in the structured text whose deterministic values ​​are lower than a preset threshold to obtain a set of high-confidence diagnostic entities.

[0188] S13. Input each diagnostic entity in the high-confidence diagnostic entity set into the pre-trained medical text encoder for semantic encoding to obtain entity semantic vectors, and input each standard term in the standard ECG terminology library into the medical text encoder for semantic encoding to obtain a standard semantic vector library.

[0189] S14. Calculate the cosine similarity between the semantic vector of each entity and all vectors in the standard semantic vector library. Select the standard term corresponding to the maximum cosine similarity as the standardized mapping result of the diagnostic entity. Keep the diagnostic entities with cosine similarity lower than the preset matching threshold as new temporary terms. Merge and deduplicate the diagnostic entities to generate structured diagnostic label data.

[0190] Furthermore, the ECG data processing module 22 is also used for:

[0191] S21. Preset at least two sets of time windows of different sizes as slice granularity. For each slice granularity, construct a three-dimensional tensor slice from all leads of ECG data points within the same time window. Map the three-dimensional tensor slice to a one-dimensional feature vector through a learnable linear projection layer to obtain the slice feature sequence at each granularity. Add independent learnable positional encoding to the slice feature sequence at each granularity.

[0192] S22. Input the slice feature sequence at each granularity into the multi-layer Transformer module for self-attention calculation. During the calculation process of the multi-layer Transformer module, extract the first output feature of the shallow network layer, the second output feature of the middle network layer, and the third output feature of the deep network layer. After concatenating the first output feature, the second output feature, and the third output feature, perform dimensionality reduction mapping through the first fully connected projection layer to obtain the enhanced slice feature sequence.

[0193] S23. Initialize learnable routing tokens for each granularity. The learnable routing token is a trainable parameter vector with the same feature dimension as the slice feature sequence. For each granularity, use the learnable routing token as the query matrix and the enhanced slice feature sequence under the granularity as the key matrix and value matrix. Calculate the attention weights through a multi-head cross-attention mechanism, and aggregate the enhanced slice feature sequence into the learnable routing token according to the attention weights to output the routing token of the granularity.

[0194] S24. Stack the routing tokens corresponding to at least two granularities in the sequence dimension to generate a token sequence. Input the token sequence into the Transformer encoder layer for self-attention calculation. After the self-attention calculation, concatenate all the routing tokens of all granularities output in the feature dimension and then fuse and transform them through the second fully connected projection layer to obtain ECG feature data.

[0195] Furthermore, the data fusion and analysis module 23 is also used for:

[0196] S31. Perform word segmentation on the diagnostic label text in the structured diagnostic label data to obtain a word index sequence, and convert the word index sequence into a word vector sequence through a word embedding layer;

[0197] S32. Input the word vector sequence into the Transformer encoder of the pre-trained language model to perform context modeling, obtain the hidden state vector corresponding to each word, and extract the output vector corresponding to the special marker as the semantic representation of the entire diagnostic label text.

[0198] S33. Perform dimensional transformation on the semantic representation to obtain the text semantic features.

[0199] Furthermore, the data fusion and analysis module 23 is also used for:

[0200] S41. Input the ECG feature data into the linear extension layer for transformation to obtain the ECG feature sequence, and use the ECG feature sequence as the encoder memory in the Transformer decoder architecture.

[0201] S42. Use the text semantic features as the initial input query vector of the decoder in the Transformer decoder architecture. In each layer of the Transformer decoder, use the decoder input query vector as the query matrix and the ECG feature sequence as the key matrix and value matrix. Calculate the interaction weight between the text semantic features and the ECG feature sequence. Based on the interaction weight, dynamically aggregate the ECG feature sequence and the text semantic features to generate the decoder output of the current layer.

[0202] S43. The feature vector output by the last decoder layer is used as the fusion feature, and the fusion feature is input into the classification head composed of fully connected layers to obtain the classification prediction result.

[0203] Furthermore, the ECG analysis model training module 24 is also used for:

[0204] S51. Perform L2 norm normalization on the ECG feature data and text semantic features respectively to obtain normalized ECG features and normalized text features.

[0205] S52. In a training batch, calculate the cosine similarity between all normalized ECG features and all normalized text features to obtain the cosine similarity matrix. The expression for cosine similarity is:

[0206]

[0207] in, Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of text features of each sample. Indicates the first in the batch ECG characteristic data of each sample Indicates the first in the batch Textual semantic features of each sample Represents L2 norm operations;

[0208] S53. Construct a multi-label vector for each sample within a batch based on the structured diagnostic label data. Calculate the label similarity between any two sample multi-label vectors to obtain a label similarity matrix. Set a positive sample threshold and a negative sample threshold. Map the positions in the label similarity matrix that are greater than the positive sample threshold to the same positions in the cosine similarity matrix and assign them a value of 1 to construct a positive sample mask matrix. Map the positions in the label similarity matrix that are less than the negative sample threshold to the same positions in the cosine similarity matrix and assign them a value of 1 to construct a negative sample mask matrix.

[0209] S54. Based on the positive sample mask matrix, the positive sample loss is calculated. The expression for the positive sample loss is:

[0210]

[0211] in, Indicates positive sample loss. Indicates batch size, Represents the positive sample mask matrix. Line 1 Column elements, This represents the sum of all elements in the positive sample mask matrix. Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of textual features among the samples;

[0212] S55. Based on the negative sample mask matrix, the negative sample loss is calculated. The expression for the negative sample loss is:

[0213]

[0214] in, Indicates negative sample loss. Indicates batch size, Represents the positive sample mask matrix. Line 1 Column elements, This represents the sum of all elements in the positive sample mask matrix. Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of text features of each sample. This represents the preset boundary value hyperparameters;

[0215] S56. The modality consistency loss is obtained by weighted summing of the positive sample loss and the negative sample loss. The expression for the modality consistency loss is:

[0216]

[0217] in, This represents the modal consistency loss. Indicates positive sample loss. Indicates negative sample loss. These are the weighting coefficients for the positive sample loss. The weighting coefficients are for the negative sample loss.

[0218] Furthermore, the ECG analysis model training module 24 is also used for:

[0219] S61. Based on the fusion features, the existence probability of each preset diagnostic category is obtained through the fully connected layer of the classification head and the sigmoid activation function;

[0220] S62. Construct a multi-label true value vector based on the structured diagnostic label data;

[0221] S63. Calculate the binary cross-entropy loss as the classification loss. The expression for the classification loss is:

[0222]

[0223] in, Represents classification loss, Represents a multi-labeled true value vector. Indicates the first In the nth sample The probability of the existence of each diagnostic category For batch size, This represents the total number of preset diagnostic categories.

[0224] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a supervised pre-training-based multimodal electrocardiogram signal representation learning method as described above.

[0225] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0226] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0227] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A supervised pre-training based multi-modal electrocardiosignal representation learning method, characterized in that, The method includes: S1. Obtain a clinical text report, perform entity extraction and standardized mapping on the clinical text report to obtain structured diagnostic label data; wherein, the structured diagnostic label data is used to characterize the diagnostic category of heart disease; S2. Obtain raw electrocardiogram (ECG) data, input the raw ECG data into a multi-granularity ECG encoder for cross-channel slicing and routing aggregation to obtain ECG feature data; wherein, the ECG feature data is used to characterize the spatiotemporal features of multi-scale ECG signals; S3. Input the structured diagnostic label data and the electrocardiogram feature data into a multimodal fusion network for text semantic extraction and cross-modal interaction to obtain text semantic features and fusion features; wherein, the fusion features are used to characterize the pathological features of electrocardiogram signals under text semantic constraints; S4. Based on the ECG feature data, the text semantic features, and the fusion features, construct modality consistency loss and classification loss, and jointly optimize the multi-granularity ECG encoder and the multi-modal fusion network based on the modality consistency loss and the classification loss to obtain the ECG analysis model; S5. Input the ECG data to be tested into the ECG analysis model for feature encoding and classification prediction to obtain the ECG analysis results.

2. The method of claim 1, wherein, S1 includes: S11. Construct prompts containing knowledge of the cardiology field based on a standard ECG terminology database and common clinical abbreviations. Then, concatenate the prompts and the clinical text report and input them into a large language model for fine-tuning the instructions to parse the clinical text report and output structured text containing diagnostic entity names and deterministic values. S12. Filter out diagnostic entities in the structured text whose deterministic values ​​are lower than a preset threshold to obtain a set of high-confidence diagnostic entities; S13. Input each of the diagnostic entities in the high-confidence diagnostic entity set into the pre-trained medical text encoder for semantic encoding to obtain entity semantic vectors, and input each standard term in the standard ECG terminology library into the medical text encoder for semantic encoding to obtain a standard semantic vector library. S14. Calculate the cosine similarity between each entity semantic vector and all vectors in the standard semantic vector library, select the standard term corresponding to the maximum cosine similarity as the standardized mapping result of the diagnostic entity, and retain the diagnostic entities with cosine similarity lower than the preset matching threshold as newly added temporary terms, merge and deduplicate the diagnostic entities, and generate the structured diagnostic label data.

3. The method according to claim 2, characterized in that, S2 includes: S21. Preset at least two sets of time windows of different sizes as slice granularities. For each slice granularity, construct a three-dimensional tensor slice from all leads of ECG data points within the same time window. Map the three-dimensional tensor slice to a one-dimensional feature vector through a learnable linear projection layer to obtain the slice feature sequence under the granularity. Add independent learnable position codes to the slice feature sequence of each granularity. S22. Input the slice feature sequence at each granularity into the multi-layer Transformer module for self-attention calculation. During the calculation process of the multi-layer Transformer module, extract the first output feature of the shallow network layer, the second output feature of the middle network layer, and the third output feature of the deep network layer. After concatenating the first output feature, the second output feature, and the third output feature, perform dimensionality reduction mapping through the first fully connected projection layer to obtain the enhanced slice feature sequence. S23. Initialize a learnable routing token for each granularity. The learnable routing token is a trainable parameter vector with the same feature dimension as the slice feature sequence. For each granularity, use the learnable routing token as the query matrix and the enhanced slice feature sequence under the granularity as the key matrix and value matrix. Calculate the attention weights through a multi-head cross-attention mechanism. Aggregate the enhanced slice feature sequence into the learnable routing token according to the attention weights and output the routing token for the granularity. S24. Stack the routing tokens corresponding to the at least two granularities in the sequence dimension to generate a token sequence. Input the token sequence into the Transformer encoder layer for self-attention calculation. After the self-attention calculation, concatenate all the routing tokens of all granularities output in the feature dimension and then perform fusion and dimension transformation through the second fully connected projection layer to obtain the ECG feature data.

4. The method according to claim 3, characterized in that, S3 includes: S31. The diagnostic label text in the structured diagnostic label data is segmented into words to obtain a word index sequence, and the word index sequence is converted into a word vector sequence through a word embedding layer; S32. Input the word vector sequence into the Transformer encoder of the pre-trained language model to perform context modeling, obtain the hidden state vector corresponding to each word, and extract the output vector corresponding to the special marker as the semantic representation of the entire diagnostic label text. S33. Perform dimensional transformation on the semantic representation to obtain the text semantic features.

5. The method according to claim 4, characterized in that, S3 further includes: S41. Input the ECG feature data into the linear extension layer for transformation to obtain the ECG feature sequence, and use the ECG feature sequence as the encoder memory in the Transformer decoder architecture. S42. Using the text semantic features as the initial input query vector of the decoder in the Transformer decoder architecture, in each layer of the Transformer decoder, using the decoder input query vector as the query matrix and the ECG feature sequence as the key matrix and value matrix, calculate the interaction weight between the text semantic features and the ECG feature sequence, dynamically aggregate the text semantic features from the ECG feature sequence according to the interaction weight, and generate the decoder output of the current layer; S43. The feature vector output by the last layer decoder is used as the fusion feature, and the fusion feature is input into the classification head composed of fully connected layers to obtain the classification prediction result.

6. The method according to claim 5, characterized in that, S4 includes: S51. Perform L2 norm normalization on the electrocardiogram feature data and the text semantic features respectively to obtain normalized electrocardiogram features and normalized text features; S52. In a training batch, calculate the cosine similarity between all normalized ECG features and all normalized text features to obtain the cosine similarity matrix. The expression for the cosine similarity is: in, Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of text features of each sample. Indicates the first in the batch ECG characteristic data of each sample Indicates the first in the batch Textual semantic features of each sample Represents L2 norm operations; S53. Construct a multi-label vector for each sample within a batch based on the structured diagnostic label data, calculate the label similarity between any two sample multi-label vectors to obtain a label similarity matrix, set a positive sample threshold and a negative sample threshold, map the positions in the label similarity matrix that are greater than the positive sample threshold to the same positions in the cosine similarity matrix and assign them a value of 1 to construct a positive sample mask matrix, map the positions in the label similarity matrix that are less than the negative sample threshold to the same positions in the cosine similarity matrix and assign them a value of 1 to construct a negative sample mask matrix; S54. Based on the positive sample mask matrix, the positive sample loss is calculated, and the expression for the positive sample loss is: in, Indicates positive sample loss. Indicates batch size, Represents the positive sample mask matrix. Line 1 Column elements, This represents the sum of all elements in the positive sample mask matrix. Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of textual features among the samples; S55. Based on the negative sample mask matrix, the negative sample loss is calculated, and the expression for the negative sample loss is: in, Indicates negative sample loss. Indicates batch size, Represents the positive sample mask matrix. Line 1 Column elements, This represents the sum of all elements in the positive sample mask matrix. Indicates the first in the batch The electrocardiogram characteristics of the sample and the first The similarity of text features of each sample. This represents the preset boundary value hyperparameters; S56. The positive sample loss and the negative sample loss are weighted and summed to obtain the modality consistency loss. The expression for the modality consistency loss is as follows: in, This represents the modal consistency loss. Indicates positive sample loss. Indicates negative sample loss. These are the weighting coefficients for the positive sample loss. The weighting coefficients are for the negative sample loss.

7. The method according to claim 6, characterized in that, S4 further includes: S61. Based on the fusion features, the existence probability of each preset diagnostic category is obtained through the fully connected layer of the classification head and the sigmoid activation function; S62. Construct a multi-label true value vector based on the structured diagnostic label data; S63. Calculate the binary cross-entropy loss as the classification loss. The expression for the classification loss is: in, Represents classification loss, Represents a multi-labeled true value vector. Indicates the first In the nth sample The probability of the existence of each diagnostic category For batch size, This represents the total number of preset diagnostic categories.

8. A multimodal electrocardiogram signal representation learning system based on supervised pre-training, characterized in that, The system includes: The text data acquisition and standardization module is used to acquire clinical text reports, perform entity extraction and standardization mapping on the clinical text reports, and obtain structured diagnostic label data; wherein, the structured diagnostic label data is used to characterize the diagnostic categories of heart diseases; The ECG data processing module is used to acquire raw ECG data, input the raw ECG data into a multi-granularity ECG encoder for cross-channel slicing and routing aggregation, and obtain ECG feature data; wherein, the ECG feature data is used to characterize the spatiotemporal features of ECG signals at multiple scales; The data fusion and analysis module is used to input the structured diagnostic label data and the electrocardiogram feature data into a multimodal fusion network for text semantic extraction and cross-modal interaction to obtain text semantic features and fusion features; wherein, the fusion features are used to characterize the pathological features of electrocardiogram signals under text semantic constraints; The ECG analysis model training module is used to construct modality consistency loss and classification loss based on the ECG feature data, the text semantic features and the fusion features, and to jointly optimize the multi-granularity ECG encoder and the multi-modal fusion network based on the modality consistency loss and the classification loss to obtain the ECG analysis model; The ECG data analysis module is used to input the ECG data to be tested into the ECG analysis model for feature encoding and classification prediction to obtain the ECG analysis results.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images