Multimodal fusion model, electronic device, and computer-readable storage medium

Abstract

The application discloses a multi-modal fusion model, an electronic device and a computer readable storage medium, comprising a first processing, a second processing and a fusion module for splicing a first feature representation and a second feature representation and forming an input vector; the fusion module comprises a gating network and a plurality of experts; the gating network is used for generating expert activation weights according to the input vector, and selecting K experts with the highest activation probability according to the weights; the outputs of the selected K experts are weighted and summed to generate a fusion feature representation; the application has the beneficial effects that: the gating network generates expert activation weights according to the input vector, and selects K experts with the highest activation probability according to the weights to generate a fusion feature representation; compared with a traditional fixed structure fusion mode, the design enhances the adaptability of the model to different input modes.

Description

Technical Field

[0001] This invention relates to the fields of computer science and biomedical engineering, specifically to multimodal fusion models, electronic devices, and computer-readable storage media. Background Technology

[0002] Cardiovascular disease is one of the leading causes of death worldwide, and its early and accurate diagnosis is crucial for improving patient outcomes and reducing healthcare costs. Currently, electrocardiogram (ECG) signals and electronic health records are commonly used clinical diagnostic tools. ECG signals are widely used due to their non-invasive and low-cost advantages, while electronic health records provide rich structured clinical information. However, existing models suffer from low efficiency in fusing these two data points. Summary of the Invention

[0003] To achieve the above objectives, the present invention provides the following technical solution: a multimodal fusion model, comprising... The first processing step is used to process time-series physiological signals and output a first feature representation. The second processing step is used to process structured clinical data and output a second feature representation. The fusion module is used to concatenate the first feature representation and the second feature representation to form an input vector; The fusion module includes a gated network and multiple experts; The gating network is used to generate expert activation weights based on the input vector, and select the K experts with the highest activation probabilities based on the weights; The outputs of the selected K experts are weighted and summed to generate a fusion feature representation; A classifier is used to process the fused feature representation and output auxiliary diagnostic results.

[0004] This invention employs a first processing method targeting time-series physiological signals and a second processing method targeting structured clinical data. In the fusion module, a multi-expert structure driven by a gating network is introduced. Expert activation weights are dynamically generated based on the input vector, and the K experts with the highest activation probabilities are selected to participate in feature fusion. Through this design, on the one hand, data from different modalities can be adaptively weighted and reconstructed at the feature level, emphasizing the feature subspaces more critical to the current task and suppressing redundant or noisy information. On the other hand, it avoids the shortcomings of traditional fixed-structure fusion methods, such as a single fusion strategy and difficulty in accommodating multiple input modes. This improves the efficiency and robustness of multimodal feature fusion, enhancing the overall recognition accuracy and generalization ability in cardiovascular disease auxiliary diagnosis tasks.

[0005] In a further optimization of the present invention, the first processing is used to process the ECG and output a first feature representation; The second processing is used to process the EHR and output a second feature representation.

[0006] This invention further specifies the first processing as feature extraction from ECG electrocardiogram signals and the second processing as feature extraction from EHR electronic health records, achieving joint modeling of the same patient's temporal physiological signals and structured clinical information. By capturing the dynamic changes in myocardial electrical activity through ECG, background information such as past medical history, medication use, and laboratory tests is characterized by EHR, and unified encoding and fusion are performed in the subsequent fusion module. This fully explores the complementary relationship between the two types of data, alleviating the problems of incomplete information and easy misjudgment under a single modality. As a result, more comprehensive and fine-grained discriminatory criteria are obtained in early screening and risk stratification tasks for cardiovascular diseases, enabling more accurate and stable prediction of patient status.

[0007] A further optimization of the present invention is that the first process includes 1D-ViT; The feedforward network layer of 1D-ViT is a MoE module; The MoE module includes gated networks, shared experts, and MLP specialists.

[0008] This invention introduces a 1D-ViT structure in the first processing module to model one-dimensional ECG sequences using a self-attention mechanism. This enables the capture of long-range dependencies and global morphological features of ECG signals over a longer time span. Simultaneously, the feedforward network layer of the 1D-ViT is designed as a MoE module, allowing different expert subnetworks to learn different types of ECG patterns under the scheduling of a gating network. By setting shared experts and MLP-specific experts within the MoE module, the shared experts learn common fundamental features of ECG signals, ensuring the stability of the representation space; while the specialized experts learn finer-grained specific features for different arrhythmia types or pathological morphologies. This design improves the model's ability to express complex ECG morphologies and its adaptability to rare pathological patterns without significantly increasing computational overhead, thereby enhancing the accuracy and robustness of ECG representation.

[0009] In a further optimization of the present invention, the MoE module includes a gated network, a shared expert, and multiple MLP specialized experts.

[0010] By limiting the MoE module to a structure of "one gating network + one shared expert + multiple MLP specialized experts," this invention achieves a better balance between the number of experts and structural complexity. The gating network can adaptively allocate activation weights to each specialized expert based on the distribution of the current input ECG features, selecting only the most relevant experts to participate in forward computation, thus significantly reducing resource waste caused by ineffective computation. The shared expert ensures the consistency of the feature space among different batches of inputs, while the majority of MLP specialized experts enhance the model's ability to distinguish between multiple ECG abnormality patterns. Compared to a single feedforward layer or a uniformly activated multi-branch structure, this MoE design improves model capacity and expressive diversity while maintaining controllable parameter counts, which is beneficial for achieving better convergence performance and generalization on large-scale ECG data.

[0011] A further optimization of the present invention includes the following second processing: The text conversion unit is used to convert structured clinical data into natural language-style text; A TF-IDF encoder is used to encode natural language style text and generate TF-IDF vectors. A multilayer perceptron is used to extract the second feature representation from the TF-IDF vector.

[0012] This invention incorporates a text conversion unit in the second processing module. This unit automatically converts structured clinical data, originally stored as numerical or text fields, into natural language-style text descriptions. A TF-IDF encoder then vectorizes this text, and a multilayer perceptron extracts a second feature representation from the TF-IDF vectors. This process allows for two main advantages: firstly, it maps discrete, multi-source clinical fields to a unified semantic space without altering the original EHR storage format, enhancing the modeling ability of potential associations between different fields and reducing reliance on complex manual feature engineering; secondly, TF-IDF highlights clinical terms with high discriminative power in the current patient record, and combined with the nonlinear mapping of the multilayer perceptron, the generated EHR feature representation maintains interpretability while possessing stronger discriminative power, thereby increasing the contribution of structured clinical data to the overall multimodal model.

[0013] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the multimodal fusion model as described above.

[0014] This invention integrates a memory and a processor into an electronic device, and pre-loads a computer program implementing the aforementioned multimodal fusion model into the memory. This allows the electronic device to automatically acquire, preprocess, fuse features, and output auxiliary diagnostic results for multimodal data such as ECG and EHR when the processor executes the program. By encapsulating the complex deep learning inference process within the electronic device, it enables auxiliary diagnosis of cardiovascular disease risks in hospital information system terminals, bedside monitoring devices, wearable terminals, or remote consultation terminals. Furthermore, it reduces manual intervention and cross-system data transfer, lowers deployment barriers and maintenance costs, and improves overall system efficiency and usability in clinical scenarios.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal fusion model as described above.

[0016] This invention stores a computer program implementing the aforementioned multimodal fusion model on a computer-readable storage medium, enabling the program to be loaded and executed by different types of processors. This allows for the unified implementation of multimodal cardiovascular disease auxiliary diagnostic functions across different hardware platforms. This design facilitates modular encapsulation and cross-platform migration of the algorithm, ensures consistency of diagnostic logic across deployment environments, shortens system development and upgrade cycles, and enhances the versatility and engineering application value of the invention.

[0017] The beneficial effects of this invention are as follows: The gated network generates expert activation weights based on the input vector, and selects the K experts with the highest activation probabilities based on the weights to generate fusion feature representations. Compared with the traditional fixed-structure fusion method, this design enhances the model's adaptability to different input patterns. Attached Figure Description

[0018] Figure 1 This is a diagram of the overall architecture of the model of the present invention.

[0019] Figure 2 This is a diagram of the ECG feature extraction branch architecture of the present invention.

[0020] Figure 3 This is a diagram of the internal structure of the Transformer-MoE of the present invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.

[0022] Reference Figure 1-3 As shown, a multimodal fusion model; Step 1: Data Preprocessing Step 1.1: Data noise processing design: To reduce noise in the raw ECG signal, we applied Butterworth filters: a low-pass filter (cutoff frequency 50 Hz) to remove power supply interference and EMG noise, and a high-pass filter (cutoff frequency 0.5 Hz) to eliminate baseline drift. This preserved key features while suppressing irrelevant artifacts.

[0023] Step 1.2: Data Feature Normalization Design: In deep learning, feature dimensions often have different scales, which can lead to training instability and performance degradation. To mitigate this, we apply Z-Score normalization, normalizing each feature to zero mean and unit variance. This process is defined as: ; Where x represents the original feature value, μ is the mean of the feature calculated on the training set, σ is the standard deviation, and z is the standardized output.

[0024] Step 2: Model Building Step 2.1: Overall Model Framework Design: A dual-branch architecture is employed to process ECG and EHR data separately, with feature fusion and joint prediction achieved through a MoE module. Specifically, the ECG branch is built upon 1D-ViT, and the feedforward layer is replaced with a MoE module. Each MoE block includes a gated network, a shared expert consisting of a convolutional network, and multiple MLP specialized experts; The EHR branch adopts a TFIDF-MLP hybrid architecture; the overall framework completes end-to-end prediction through dual-branch feature extraction and MoE fusion modules.

[0025] Step 2.2: ECG Feature Extraction Branch Design: We employ 1D-ViT to extract temporal features from the original ECG, given the input sequence. We divide it into N = [L / P] non-overlapping blocks, each block having a length of P. Encoded by linear projection and position as: ; ; in It is a learnable projection matrix. It is a bias term. It is the encoding of the i-th position.

[0026] For each patch, linear projection and positional encoding are performed, and all patch embeddings are concatenated to form the Transformer input sequence: ; Each Transformer block applies multi-head self-attention to model local and global dependencies. The scaled dot-product attention is calculated as follows: ; To improve model capacity and controllable computation, we replaced the original feedforward network (FFN) with a MoE layer. Each MoE block consists of E experts: one shared expert. and E-1 specialists The shared expert processes each token through two one-dimensional convolutional layers: ; in It is a token embedding passed to the expert layer.

[0027] For specialized experts, the gated network computes sparse top-K selection weights: ; ; in It is a gating matrix. It is the activation weight. This indicates the selected expert indicator.

[0028] The final output of the MoE layer combines the outputs of two experts: ; To promote balanced utilization of all experts, we introduce a load balancing loss. The average activation probability of expert j in a batch of M tokens is: ; The corresponding load loss is: ; Finally, the global ECG representation is calculated by average pooling the Transformer output tokens: ; in It is the first after the MoE-enhanced Transformer block. The output embedding of each token.

[0029] Step 2.3: EHR Feature Extraction Branch Design: To address the limitations of structured EHR data, such as limited semantic expressiveness, lack of contextual information, and difficulty in capturing potential feature associations, we convert EHRs into natural language-style text and encode them using the TF-IDF algorithm, forming a TF-IDF-MLP module. TF-IDF highlights discriminative medical terms by combining term frequency and inverse document frequency, and is defined as follows: ; in Indicates sample Chinese terminology frequency, It includes The number of samples, This represents the total number of samples. The logarithmic term acts as the inverse document frequency, reducing the weight of frequent terms and highlighting informational content.

[0030] Each EHR text is represented as a sparse vector. The vector is used to extract semantic features through a three-layer MLP with LeakyReLU activation. ; in and It is the first The weight matrix and bias vector of each MLP layer This represents the LeakyReLU activation, where D is the output dimension aligned with the ECG branch. Then, the resulting dense representation is... Used in multimodal fusion modules.

[0031] Step 2.4: Design of MoE Feature Fusion and Prediction Module: To achieve semantic interaction and task-level coordination between modalities, we designed a fused MoE module to dynamically model features from the ECG and EHR branches. The final representation from both modalities is as follows: ; We first concatenate the features of the two modalities as the fusion input: ; Then, By fusing the MoE layer, a gating mechanism selects a subset of experts for processing. Let the fusing MoE layer contain E experts. And the gating weights are defined as: ; We select the K experts with the highest activation probabilities to form a subset. And a weighted fusion method with normalized weights is adopted: ; in It is a gating parameter matrix. Indicates the first A fusion expert, This is the obtained cross-modal representation.

[0032] Finally, the fused representation is passed to the classifier to produce the final prediction: ; in, and These are classifier parameters. It is the sigmoid activation function, and It is the predicted score.

[0033] During training, the entire network is optimized using a combined loss consisting of the main task loss and the load balancing loss of all MoE modules.

[0034] ; in, These are weighting coefficients. Indicates the first Load balancing losses across MoE layers, including ECG layering and fused MoE layers.

[0035] Step 3: Obtain the offline model: A usable model is obtained by training the source dataset; Step 4: Online model testing: The model obtained in step 3 is used to test the test set of the target dataset, and the results are diagnosed.

[0036] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal fusion model, characterized in that, include The first processing step is used to process time-series physiological signals and output a first feature representation. The second processing step is used to process structured clinical data and output a second feature representation. The fusion module is used to concatenate the first feature representation and the second feature representation to form an input vector; The fusion module includes a gated network and multiple experts; The gating network is used to generate expert activation weights based on the input vector, and select the K experts with the highest activation probabilities based on the weights; The outputs of the selected K experts are weighted and summed to generate a fusion feature representation; A classifier is used to process the fused feature representation and output auxiliary diagnostic results.

2. The multimodal fusion model according to claim 1, characterized in that, The first processing is used to process the ECG and output a first feature representation; The second processing is used to process the EHR and output a second feature representation.

3. The multimodal fusion model according to claim 1, characterized in that, The first process includes 1D-ViT; The feedforward network layer of 1D-ViT is a MoE module; The MoE module includes gated networks, shared experts, and MLP specialists.

4. The multimodal fusion model according to claim 3, characterized in that, The MoE module includes a gated network, a shared expert, and multiple MLP specialized experts.

5. The multimodal fusion model according to claim 1, characterized in that, The second process includes: The text conversion unit is used to convert structured clinical data into natural language-style text; A TF-IDF encoder is used to encode natural language style text and generate TF-IDF vectors. A multilayer perceptron is used to extract the second feature representation from the TF-IDF vector.

6. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes a computer program, it implements the multimodal fusion model as described in any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal fusion model as described in any one of claims 1-5.

Images