A method and system for chest pain classification based on multimodal data fusion and deep learning models

Abstract

This invention relates to a chest pain classification method and system based on multimodal data fusion and a deep learning model, belonging to the field of intelligent medical auxiliary diagnosis technology. First, chest X-ray images undergo cardiac region segmentation and pathological feature enhancement preprocessing. An optimized multi-scale visual model is used to extract anatomical features. Simultaneously, multi-lead temporal calibration and ST segment waveform labeling are performed on electrocardiogram signals. Electrophysiological temporal features are captured through a multi-scale one-dimensional convolutional network. Subsequently, a dynamically gated hybrid expert system is constructed, in which anatomical experts, electrophysiological experts, multimodal association experts, and critical value experts each process specific modal features. The gating network dynamically calculates expert weights based on real-time vital signs and pathological features. This invention, by integrating multimodal data and combining deep learning technology, can provide clinicians with rapid and accurate chest pain classification support and report generation, and has broad application prospects.

Description

Technical Field

[0001] This invention relates to a chest pain classification method and system based on multimodal data fusion and deep learning models, belonging to the field of intelligent medical auxiliary diagnosis technology. Background Technology

[0002] Chest pain is a common and important symptom in thoracic surgery. Its occurrence often involves multiple underlying causes, including but not limited to coronary heart disease, myocardial infarction, lung disease, and other pleural diseases. Accurate diagnosis of chest pain is crucial for timely management and treatment, especially in cases of acute chest pain, where rapid identification of the cause directly impacts patient safety and treatment outcomes. Currently, the classification and diagnosis of chest pain typically rely on the physician's clinical experience and a range of traditional diagnostic methods, such as electrocardiogram (ECG), chest X-ray, computed tomography (CT) scan, and laboratory tests. While these methods provide valuable clinical information, they often have limitations. First, the clinical manifestations of chest pain can be highly similar, and symptoms from different causes may not be clearly distinguishable. Therefore, relying solely on the physician's experience or a single test result makes it difficult to accurately determine the cause in a short time. Second, traditional imaging and testing methods rely on manual analysis, the results of which are limited by the physician's expertise and workload, potentially leading to missed or misdiagnosed cases. Furthermore, certain acute heart diseases, such as myocardial infarction, require early diagnosis and emergency treatment; therefore, rapid diagnosis of the cause of chest pain is crucial for improving patient survival rates.

[0003] With the rapid development of artificial intelligence (AI) technology, especially the widespread application of deep learning in medical imaging and signal processing, research on chest pain classification is gradually moving towards automation. Deep learning can extract complex feature information from medical images such as chest X-rays and electrocardiograms, and perform analysis and classification, overcoming the limitations of traditional manual diagnosis. In recent years, an increasing number of studies have attempted to use deep learning technologies, particularly models such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), to automate the analysis of chest imaging and electrocardiogram data, identifying the causes and types of chest pain. Deep learning-based automated diagnostic systems can significantly improve the efficiency and accuracy of chest pain diagnosis, providing real-time auxiliary decision support for clinical practice.

[0004] However, despite the significant progress deep learning has made in medical imaging and signal processing, existing methods still face several challenges. First, most existing methods focus primarily on single-modal data processing, failing to effectively integrate information from multiple sources (such as chest X-rays and electrocardiogram data), thus limiting the accuracy and robustness of classification results. Second, the training of existing deep learning models largely relies on large amounts of labeled data, while chest pain classification tasks often face data scarcity or imbalance, leading to poor model performance when dealing with some more specific causes of chest pain. More importantly, although deep learning technology excels in classification accuracy, it lacks sufficient interpretability, which is particularly critical for medical applications. Clinicians need to be able to understand the decisions made by the model in order to trust and adopt the system's recommendations in complex medical situations. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a chest pain classification method and system based on multimodal data fusion and a deep learning model. First, chest X-ray images undergo cardiac region segmentation and pathological feature enhancement preprocessing. Anatomical features are extracted using a multi-scale visual model (Vit-Med) optimized for chest X-rays. Simultaneously, multi-lead temporal calibration and ST-segment waveform labeling are performed on electrocardiogram signals. Electrophysiological temporal features are captured through a multi-scale one-dimensional convolutional network. Subsequently, a dynamic gating hybrid expert system (DG-MOE) is constructed, where anatomical experts, electrophysiological experts, multimodal association experts, and critical value experts handle specific modal features. The gating network dynamically calculates expert weights based on real-time vital signs (such as heart rate and blood pressure) and pathological features (such as ST-segment offset).

[0006] The technical solution of the present invention is as follows:

[0007] A chest pain classification method based on multimodal data fusion and deep learning models, comprising the following steps:

[0008] (1) Preprocessing and enhancing chest X-ray images and electrocardiogram multimodal data;

[0009] (2) Anatomical-temporal dual-stream feature extraction;

[0010] (3) Clinical knowledge guides the dynamic integration of MOE;

[0011] (4) Diagnostic reasoning driven by large models.

[0012] According to a preferred embodiment of the present invention, step (1) specifically includes the following steps:

[0013] To address the heterogeneity and noise issues of medical multimodal data, this step constructs a preprocessing pipeline guided by clinical prior knowledge. Chest X-ray images are preprocessed, and key anatomical regions, including the heart and lung fields, are segmented using a pre-trained U-Net network to generate binary masks that guide ROI standardization. Adaptive window width and level adjustment (window level 40HU, window width 400HU) is employed to enhance the contrast of mediastinal structures, and pathological data preservation enhancements (±15° rotation, elastic deformation of lung field texture) are combined to improve model robustness.

[0014] Image preprocessing of electrocardiograms (ECGs): 12-lead signals were aligned based on dynamic time warping (DTW), and baseline drift was eliminated by wavelet threshold denoising (db6 wavelet, 5-level decomposition). Key ST segment / T wave waveforms were detected using an LSTM-CRF model to generate a time-series labeling matrix (e.g., marking the start / end of the ST segment), and elastic time warping (ETW) enhancement technology was applied to simulate heart rate variability. After preprocessing, the data were stored in tensor form: 512×512×1 for chest X-rays and 12-lead × 2560 sampling points for ECGs.

[0015] According to a preferred embodiment of the present invention, step (2) specifically involves the following process:

[0016] (21) A multi-scale anatomical perception pyramid network is constructed based on the anatomical features of chest X-ray images. Global and local features are extracted from the chest X-ray images, and cross-block attention fusion is performed. Then, attention enhancement is applied to the pathological regions of the chest X-ray images, and medical semantic location encoding is performed on the chest X-ray images.

[0017] (22) For electrocardiogram, a time-series feature extraction is constructed. A multi-scale time-series convolutional pyramid network is constructed to extract features from three aspects: short-time feature extraction, medium-time feature extraction, and long-time feature modeling. Then, feature fusion is performed, and a total loss function is constructed.

[0018] According to a preferred embodiment of the present invention, step (21) specifically includes the following steps:

[0019] A multi-scale anatomical perception pyramid network was constructed to target the anatomical features of chest X-ray images. Global feature extraction was performed on the chest X-ray images, which were then segmented into 16×16 pixel blocks (each block being a 256×256 region) and input into a standard VisionTransformer (ViT) module. The overall structure of the thoracic cavity (such as the heart outline and lung field distribution) was captured through a multi-head self-attention mechanism. Then, local feature extraction was performed on the chest X-ray images, and dilated convolution (dilation=2) was used to process 8×8 pixel blocks (each block being a 128×128 region) to enhance the capture of detailed features such as vascular texture and calcifications.

[0020] Cross-Patch Attention Fusion: Global features (16×16 blocks) and local features (8×8 blocks) are concatenated, and features of different scales are fused through the Cross-Patch Attention (CPA) module. The formula is as follows:

[0021]

[0022] in, The query matrix for global features. M is a key-value matrix of local features. ROI ∈{0,1} N×M For the anatomical mask matrix, only feature interactions are allowed in the heart / lung field regions, d k =d / h is the dimension of each attention head. Through multi-scale fusion, it can capture global anatomical structures and local pathological details at the same time, and improve the sensitivity to small lesions (such as 2mm lung nodules).

[0023] Enhanced attention to pathological regions in chest X-ray images, anatomical mask generation, and pre-trained U-Net segmentation network output binary masks M for key regions such as the heart and hilum. patho ∈{0,1} H×W Attention bias injection, in self-attention computation, uses the anatomical mask as a bias term to enhance the model's focus on pathological regions:

[0024]

[0025] in, λ is a block-level mask matrix. If the block contains a pathological region, it is set to 1; otherwise, it is set to 0. λ is a learnable parameter with an initial value of 0.5. It adjusts the attention intensity of the pathological region through backpropagation. By enhancing the attention to the pathological region, the model’s attention to areas such as lung field texture is improved.

[0026] Medical semantic location coding is performed on chest X-ray images. Six anatomical landmarks (such as the apex of the heart c1 = (0.42, 0.67), the aortic arch apex c2 = (0.38, 0.45), etc.) are predefined as anatomical key points. Weighted coding is performed on the inverse distance, and the distance between the center coordinates p = (x, y) of each image block and the anatomical key points is calculated to generate semantic location codes.

[0027]

[0028] Among them, c k ∈[0,1} 2 The normalized coordinates of the k-th key point are... The learnable weight matrix maps the distance to the feature space, ||pc k||2 represents the Euclidean distance, which measures the proximity of an image patch to anatomical landmarks.

[0029] According to a preferred embodiment of the present invention, step (22) specifically includes the following steps:

[0030] For electrocardiogram (ECG) temporal feature extraction (MS-TCN module), a multi-scale temporal convolutional pyramid network is constructed to extract features from three aspects: short-term feature extraction, mid-term feature extraction, and long-term feature modeling. Short-term feature extraction uses a 5ms convolutional kernel (kernel size = 128 sampling points) to capture QRS wave details (e.g., R wave peak, Q wave depth); mid-term feature extraction uses a 15ms convolutional kernel (kernel size = 384 sampling points) to analyze ST segment morphological changes (e.g., elevation / depression slope); long-term feature modeling uses a bidirectional LSTM network (128-dimensional hidden layers) to capture rhythmic features such as heart rate variability (HRV). Based on this, feature fusion is performed, concatenating the multi-scale features and then reducing the dimensionality through a 1×1 convolution, as shown in the formula:

[0031] F fused =Conv1D 1×1 (Concat(F short F mid F long (4)

[0032] in, We generate feature tensors for different time scales, then construct a clinical waveform constraint loss function to align the ST segments and constrain the ST segment features predicted by the model. ST The actual value as indicated by the doctor The error is:

[0033]

[0034] in, For the feature segment ST of the i-th sample output by the model, To ensure the waveform classification consistency of the true ST segment features extracted from the waveform annotation matrix (annotation interval ±20ms), waveform classification consistency constraints are applied to the features. KL divergence is used to ensure the waveform probability distribution p output by the model is consistent. wave Distribution of annotations Consistency:

[0035]

[0036] in, The model predicts the QRS / ST / T wave probabilities. The actual waveform distribution is labeled;

[0037] Construct a total loss function to jointly optimize feature alignment and distribution consistency:

[0038]

[0039] Where α = 0.1 is the weight for balancing the two losses. The loss value for segment ST. This represents the loss value between the image and the annotation.

[0040] According to a preferred embodiment of the present invention, step (3) specifically involves the following process:

[0041] (31) Construct four specialized expert models to handle specific modalities or tasks respectively;

[0042] (32) Construct a dynamic gating network;

[0043] (33) Perform multimodal feature fusion and splice the features of multiple expert models.

[0044] According to a preferred embodiment of the present invention, step (31) specifically comprises: constructing an anatomical expert model based on a 3D-ResNet50 network structure, inputting multi-scale features (768 dimensions) from chest X-rays, and outputting anatomical abnormality scores (such as cardiac hypertrophy index and lung field translucency), for example, activating when mediastinal widening (>3cm) or pulmonary edema is detected on chest X-rays; and constructing an electrophysiological expert model based on Transformer-ECG, inputting ECG temporal features (256 dimensions), and outputting electrophysiological parameters such as ST segment offset and QRS complex width. The parameters are set to be forcibly activated when the ST segment offset of any ECG lead is ≥1mm; a multimodal association expert model is constructed based on a graph attention network (GAT) to model the cross-modal association between chest X-ray regions (nodes) and ECG waveforms (edges), and is activated when the confidence difference between chest X-ray and ECG features is >20%; a critical value expert model is constructed based on a lightweight CNN (4-layer convolution), which takes vital signs (heart rate, blood pressure) and pathological markers (such as troponin) as input, and is forcibly activated when the heart rate is >120 beats / min or the systolic blood pressure is <90mmHg.

[0045] According to a preferred embodiment of the present invention, step (32) specifically comprises:

[0046] The gating network dynamically adjusts expert weights based on real-time clinical data, and consists of two parts: data-driven and rule-driven. The first part is data-driven weight calculation, which involves inputting multimodal features h. in and vital signs (Heart rate, blood pressure, blood oxygen), construct the gating function:

[0047]

[0048] Among them, W g The weights are dynamically calculated, with σ being a variable parameter. Then, rules drive weight adjustments, and hard constraints are applied when a specific clinical event is detected.

[0049]

[0050] The gating function varies under different constraints. When ST elevation is greater than or equal to 2 mm and k = 2, the gating constraint is 0.7; when the heart rate is greater than 120 and k = 4, the gating constraint is 0.5; and in other cases, it is [missing value]. Then, the two weights are combined using a synthesis method:

[0051]

[0052] Where σ is the Sigmoid function, which restricts the weights to the interval [0,1], and α is a learnable parameter that balances the contributions of data and rules (initial value 0.6). This is the weight matrix of the gated network.

[0053] According to a preferred embodiment of the present invention, step (33) specifically comprises:

[0054] Multimodal feature fusion is performed by concatenating features from multiple expert models using residual connections.

[0055]

[0056] h final =h in +W r ·h expert (12)

[0057] Among them, E k (·) represents the feature extraction function of expert k. The residual projection matrix is... The final fused feature vector has a dimension of 1024.

[0058] According to a preferred embodiment of the present invention, step (4) specifically involves the following process:

[0059] Construct an input feature concatenation method to combine the 1024-dimensional h output in step (3). final The feature vector is the 768-dimensional vector t encoded from the patient's electronic medical record text (chief complaint, medical history, etc.). text To splice:

[0060] h joint =W c ·Concat(h final , t text (13)

[0061] in, Given a dimension-reduced matrix (1792 = 1024 + 768), use Concat as the concatenation method; the output is...

[0062] Diagnostic decision generation uses causal attention decoding to generate diagnostic content and LLaMA-Med's Transformer decoder to generate the diagnostic logic chain. Attention computation incorporates medical knowledge constraints.

[0063]

[0064] Among them, M rule ∈{0,1} L×L Let L be the clinical rule mask matrix, and d be the sequence length. k =2048 / 32=64 is the dimension of each attention head;

[0065] Structured output generation: The model output contains three parallel headers: a disease probability header, a treatment suggestion header, and a differential diagnosis header. The disease probability header is as follows:

[0066] p disease =Softmax(W p ·h in (15)

[0067] in, The classification is weighted into four categories (pulmonary embolism, coronary artery disease, normal, and aortic class).

[0068] The suggested handling header is:

[0069] s action =Sigmoid(W a ·h in (16)

[0070] in, The procedure includes 12 standard treatments (such as "sublingual administration of nitroglycerin" and "activation of the catheterization lab").

[0071] Differential diagnosis of head is:

[0072]

[0073] Where, ε DDx An embedding vector library containing 30 chest pain-related diseases.

[0074] Generate a natural language decision chain based on the above steps:

[0075] Generate report templates that align with clinical reasoning.

[0076] A chest pain classification system based on multimodal data fusion and deep learning models, comprising:

[0077] The preprocessing module is used to preprocess and enhance chest X-ray images and electrocardiogram multimodal data;

[0078] Extraction module for anatomical-temporal dual-stream feature extraction;

[0079] The fusion module is used for dynamic fusion of clinical knowledge-guided MOEs.

[0080] The inference module is used for diagnostic inference driven by large models.

[0081] The beneficial effects of this invention are as follows:

[0082] This invention combines multimodal data such as chest X-rays and electrocardiograms (ECGs). It uses a dynamic gating network to analyze the anatomical features of chest X-rays and the temporal characteristics of ECGs in real time. It adaptively adjusts expert weights based on vital signs (e.g., increasing the weight of electrophysiology experts to 70% when ST-segment elevation occurs), utilizes multimodal contrastive learning to achieve cross-modal semantic alignment, and embeds clinical rule constraints (e.g., triggering critical value experts when heart rate >120 bpm) to ensure the compliance of diagnostic logic. Simultaneously, it integrates a large medical model to generate an interpretable report containing bimodal heatmaps and natural language reasoning chains. This method, by integrating multimodal data and combining deep learning technology, can provide clinicians with rapid and accurate chest pain classification support and report generation, and has broad application prospects. Attached Figure Description

[0083] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0084] The present invention will be further described below with reference to the embodiments and accompanying drawings, but is not limited thereto.

[0085] Example 1:

[0086] This implementation provides a chest pain classification method based on multimodal data fusion and a deep learning model, with the following steps:

[0087] (1) Preprocessing and enhancement of chest X-ray images and electrocardiogram multimodal data, the specific steps are as follows:

[0088] To address the heterogeneity and noise issues of medical multimodal data, this step constructs a preprocessing pipeline guided by clinical prior knowledge. Chest X-ray images are preprocessed, and key anatomical regions, including the heart and lung fields, are segmented using a pre-trained U-Net network to generate binary masks that guide ROI standardization. Adaptive window width and level adjustment (window level 40HU, window width 400HU) is employed to enhance the contrast of mediastinal structures, and pathological data preservation enhancements (±15° rotation, elastic deformation of lung field texture) are combined to improve model robustness.

[0089] Image preprocessing of electrocardiograms (ECGs): 12-lead signals were aligned based on dynamic time warping (DTW), and baseline drift was eliminated by wavelet threshold denoising (db6 wavelet, 5-level decomposition). Key ST segment / T wave waveforms were detected using an LSTM-CRF model to generate a time-series labeling matrix (e.g., marking the start / end of the ST segment), and elastic time warping (ETW) enhancement technology was applied to simulate heart rate variability. After preprocessing, the data were stored in tensor form: 512×512×1 for chest X-rays and 12-lead × 2560 sampling points for ECGs.

[0090] (2) Anatomical-temporal dual-stream feature extraction, the specific process is as follows:

[0091] (21) A multi-scale anatomical perception pyramid network was constructed based on the anatomical features of chest X-ray images. Global features were extracted from the chest X-ray images, and the chest X-ray images were divided into 16×16 pixel blocks (each block is a 256×256 region). The blocks were then input into the standard VisionTransformer (ViT) module. The overall structure of the thoracic cavity (such as the heart outline and lung field distribution) was captured through a multi-head self-attention mechanism. Then, local features were extracted from the chest X-ray images. Dilated convolution (dilation=2) was used to process 8×8 pixel blocks (each block is a 128×128 region) to enhance the ability to capture detailed features such as vascular texture and calcification points.

[0092] Cross-Patch Attention Fusion: Global features (16×16 blocks) and local features (8×8 blocks) are concatenated, and features of different scales are fused through the Cross-Patch Attention (CPA) module. The formula is as follows:

[0093]

[0094] in, The query matrix for global features. M is a key-value matrix of local features. ROI ∈{0,1} N×M For the anatomical mask matrix, only feature interactions are allowed in the heart / lung field regions, d k =d / h is the dimension of each attention head. Through multi-scale fusion, it can capture global anatomical structures and local pathological details at the same time, and improve the sensitivity to small lesions (such as 2mm lung nodules).

[0095] Enhanced attention to pathological regions in chest X-ray images, anatomical mask generation, and pre-trained U-Net segmentation network output binary masks M for key regions such as the heart and hilum. patho ∈{0,1} H×W Attention bias injection, in self-attention computation, uses the anatomical mask as a bias term to enhance the model's focus on pathological regions:

[0096]

[0097] in, λ is a block-level mask matrix. If the block contains a pathological region, it is set to 1; otherwise, it is set to 0. λ is a learnable parameter with an initial value of 0.5. It adjusts the attention intensity of the pathological region through backpropagation. By enhancing the attention to the pathological region, the model’s attention to areas such as lung field texture is improved.

[0098] Medical semantic location coding is performed on chest X-ray images. Six anatomical landmarks (such as the apex of the heart c1 = (0.42, 0.67), the aortic arch apex c2 = (0.38, 0.45), etc.) are predefined as anatomical key points. Weighted coding is performed on the inverse distance, and the distance between the center coordinates p = (x, y) of each image block and the anatomical key points is calculated to generate semantic location codes.

[0099]

[0100] Among them, c k ∈[0,1] 2 The normalized coordinates of the k-th key point are... The learnable weight matrix maps the distance to the feature space, ||pc k ||2 represents the Euclidean distance, which measures the proximity of an image patch to anatomical landmarks;

[0101] (22) For electrocardiogram (ECG), a time-series feature extraction (MS-TCN module) is constructed. A multi-scale temporal convolutional pyramid network is built to extract features from three aspects: short-time feature extraction, mid-time feature extraction, and long-time feature modeling. Short-time feature extraction uses a 5ms convolutional kernel (kernel size = 128 sampling points) to capture QRS wave details (such as R wave peak and Q wave depth); mid-time feature extraction uses a 15ms convolutional kernel (kernel size = 384 sampling points) to analyze ST segment morphological change trends (such as elevation / depression slope); long-time feature modeling uses a bidirectional LSTM network (128-dimensional hidden layers) to capture rhythm features such as heart rate variability (HRV). Based on this, feature fusion is performed. The multi-scale features are concatenated and then dimensionality is reduced by 1×1 convolution. The formula is as follows:

[0102] F fused =Conv1D 1×1 (Concat(F short F mid F long (4)

[0103] in, We generate feature tensors for different time scales, then construct a clinical waveform constraint loss function to align the ST segments and constrain the ST segment features predicted by the model. STThe actual value as indicated by the doctor The error is:

[0104]

[0105] in, For the feature segment ST of the i-th sample output by the model, To ensure the waveform classification consistency of the true ST segment features extracted from the waveform annotation matrix (annotation interval ±20ms), waveform classification consistency constraints are applied to the features. KL divergence is used to ensure the waveform probability distribution p output by the model is consistent. wave Distribution of annotations Consistency:

[0106]

[0107] in, The model predicts the QRS / ST / T wave probabilities. The actual waveform distribution is labeled;

[0108] Construct a total loss function to jointly optimize feature alignment and distribution consistency:

[0109]

[0110] Where α = 0.1 is the weight for balancing the two losses. The loss value for segment ST. This represents the loss value between the image and the annotation.

[0111] (3) Clinical knowledge guides the dynamic integration of MOE, the specific process of which is as follows:

[0112] (31) Construct four specialized expert models to handle specific modalities or tasks respectively;

[0113] An anatomical expert model was constructed based on a 3D-ResNet50 network structure. It inputs multi-scale features from chest X-rays (768 dimensions) and outputs anatomical abnormality scores (such as cardiac hypertrophy index and lung field translucency). For example, it is activated when mediastinal widening (>3cm) or pulmonary edema is detected on chest X-ray. An electrophysiological expert model was also constructed based on Transformer-ECG. It inputs ECG temporal features (256 dimensions) and outputs electrophysiological parameters such as ST segment shift and QRS complex width. The model is activated when any ECG lead... Forced activation when ST segment offset ≥1mm; Construct a multimodal association expert model based on graph attention network (GAT) to model the cross-modal association between chest X-ray region (node) and ECG waveform (edge), and activate when the confidence difference between chest X-ray and ECG features >20%; Construct a critical value expert model based on lightweight CNN (4-layer convolution), input vital signs (heart rate, blood pressure) and pathological markers (such as troponin), and force activation when heart rate >120 beats / min or systolic blood pressure <90mmHg.

[0114] (32) Construct a dynamic gating network;

[0115] The gating network dynamically adjusts expert weights based on real-time clinical data, and consists of two parts: data-driven and rule-driven. The first part is data-driven weight calculation, which involves inputting multimodal features h. in and vital signs (Heart rate, blood pressure, blood oxygen), construct the gating function:

[0116]

[0117] Among them, W g The weights are dynamically calculated, with σ being a variable parameter. Then, rules drive weight adjustments, and hard constraints are applied when a specific clinical event is detected.

[0118]

[0119] The gating function varies under different constraints. When ST elevation is greater than or equal to 2 mm and k = 2, the gating constraint is 0.7; when the heart rate is greater than 120 and k = 4, the gating constraint is 0.5; and in other cases, it is [missing value]. Then, the two weights are combined using a synthesis method:

[0120]

[0121] Where σ is the Sigmoid function, which restricts the weights to the interval [0,1], and α is a learnable parameter that balances the contributions of data and rules (initial value 0.6). This is the weight matrix of the gated network.

[0122] (33) Perform multimodal feature fusion, concatenate features from multiple expert models, and perform concatenation based on residual connections:

[0123]

[0124] h final =h in +W r ·h expert (12)

[0125] Among them, E k (·) represents the feature extraction function of expert k. The residual projection matrix is... The final fused feature vector has a dimension of 1024.

[0126] (4) Diagnostic reasoning driven by large models.

[0127] Construct an input feature concatenation method to combine the 1024-dimensional h output in step (3). final The feature vector is the 768-dimensional vector t encoded from the patient's electronic medical record text (chief complaint, medical history, etc.). text To splice:

[0128] h joint =W c ·Concat(h final , t text (13)

[0129] in, Given a dimension-reduced matrix (1792 = 1024 + 768), use Concat as the concatenation method; the output is...

[0130] Diagnostic decision generation uses causal attention decoding to generate diagnostic content and LLaMA-Med's Transformer decoder to generate the diagnostic logic chain. Attention computation incorporates medical knowledge constraints.

[0131]

[0132] Among them, M rule ∈{0,1} L×L Let L be the clinical rule mask matrix, and d be the sequence length. k =2048 / 32=64 is the dimension of each attention head;

[0133] Structured output generation: The model output contains three parallel headers: a disease probability header, a treatment suggestion header, and a differential diagnosis header. The disease probability header is as follows:

[0134] p disease=Softmax(W p ·h in (15)

[0135] in, The classification is weighted into four categories (pulmonary embolism, coronary artery disease, normal, and aortic class).

[0136] The suggested handling header is:

[0137] s action =Sigmoid(W a ·h in (16)

[0138] in, The procedure includes 12 standard treatments (such as "sublingual administration of nitroglycerin" and "activation of the catheterization lab").

[0139] Differential diagnosis of head is:

[0140]

[0141] Where, ε DDx An embedding vector library containing 30 chest pain-related diseases.

[0142] Generate a natural language decision chain based on the above steps:

[0143] Generate report templates that align with clinical reasoning.

[0144] [Key Findings]

[0145] -ECG: ST segment elevation of 2.3 mm in leads V1-V4 (electrophysiology expert weight 0.82)

[0146] - Chest X-ray: Positive hilar butterfly sign (anatomical expert weight 0.71)

[0147] [Diagnostic Conclusion]

[0148] - Acute anterior wall myocardial infarction (94.7%), TIMI risk score 5.

[0149] [Recommendations for Handling the Issue]

[0150] 1. Immediately activate the catheterization lab (98% confidence level)

[0151] 2. Chew 300mg of aspirin (95% confidence level)

[0152] [Differential Diagnosis]

[0153] - Pulmonary embolism (3.2%): D-dimer negative result can rule it out.

[0154] Ultimately, by deeply integrating the semantic understanding capabilities of the large model with the medical knowledge base, intelligent mapping from multimodal data to clinical decisions was achieved, ensuring high accuracy while meeting the timeliness requirements of emergency scenarios.

[0155] Example 2:

[0156] A chest pain classification system based on multimodal data fusion and deep learning models, comprising:

[0157] The preprocessing module is used to preprocess and enhance chest X-ray images and electrocardiogram multimodal data;

[0158] Extraction module for anatomical-temporal dual-stream feature extraction;

[0159] The fusion module is used for dynamic fusion of clinical knowledge-guided MOEs.

[0160] The inference module is used for diagnostic inference driven by large models.

Claims

1. A chest pain classification method based on multimodal data fusion and deep learning models, characterized in that, The steps are as follows: (1) Preprocessing and enhancing chest X-ray images and electrocardiogram multimodal data; (2) Anatomical-temporal dual-stream feature extraction, the specific process is as follows: (21) Construct a multi-scale anatomical perception pyramid network for the anatomical features of chest X-ray images, extract global and local features from chest X-ray images, perform cross-block attention fusion, enhance attention to pathological regions of chest X-ray images, and encode medical semantic location of chest X-ray images. (22) For electrocardiogram, construct a time-series feature extraction method, construct a multi-scale time-series convolutional pyramid network, extract features from three aspects: short-time feature extraction, medium-time feature extraction, and long-time feature modeling, then perform feature fusion, and construct a total loss function; (3) Clinical knowledge guides the dynamic integration of MOE, the specific process of which is as follows: (31) Construct an anatomical expert model based on the 3D-ResNet50 network structure, input multi-scale features of chest X-ray, and output anatomical abnormality score; construct an electrophysiological expert model based on Transformer-ECG, input ECG temporal features, and output electrophysiological parameters ST segment offset and QRS complex width, and forcibly activate when the ST segment offset of any ECG lead is ≥1mm; construct a multimodal association expert model based on graph attention network, model cross-modal association between chest X-ray region and ECG waveform, and activate when the confidence difference between chest X-ray and ECG features is >20%; construct a critical value expert model based on lightweight CNN, input vital signs and pathological markers, and forcibly activate when heart rate is >120 beats / min or systolic blood pressure is <90mmHg; (32) Construct a dynamic gating network; (33) Perform multimodal feature fusion and splice the features of multiple expert models; (4) Diagnostic reasoning driven by large models.

2. The chest pain classification method based on multimodal data fusion and deep learning model as described in claim 1, characterized in that, In step (1), the specific steps are as follows: Chest X-ray images are preprocessed, and key anatomical regions, including the heart and lung fields, are segmented using a pre-trained U-Net network to generate binary masks. Adaptive window width and level adjustment is used to enhance the contrast of mediastinal structures, and pathological data is combined to enhance the robustness of the model. Image preprocessing of electrocardiograms: Based on dynamic time warping to align 12-lead signals, wavelet thresholding is used to eliminate baseline drift; key ST segment / T wave waveforms are detected by LSTM-CRF model to generate time-series labeling matrix, and elastic time warp enhancement technology is applied to simulate heart rate variability. After preprocessing, the data is stored in tensor form.

3. The chest pain classification method based on multimodal data fusion and deep learning model as described in claim 2, characterized in that, In step (21), the specific steps are as follows: A multi-scale anatomical perception pyramid network was constructed based on the anatomical features of chest X-ray images. Global features were extracted from the chest X-ray images, which were then segmented into 16×16 pixel blocks and input into a standard Vision Transformer (ViT) module. The overall structure of the thoracic cavity was captured through a multi-head self-attention mechanism. Local features were then extracted from the chest X-ray images, and dilated convolution was used to process the 8×8 pixel blocks. Cross-block attention fusion: Global and local features are concatenated, and features at different scales are fused through a cross-block attention module. The formula is as follows: （1） in, The query matrix for global features. The key and value matrix is ​​a local feature. This is an anatomical mask matrix that only allows feature interactions within the heart / lung field regions. For each attention head dimension, multi-scale fusion is used to simultaneously capture global anatomical structures and local pathological details; Enhanced attention to pathological regions in chest X-ray images, anatomical mask generation, and the pre-trained U-Net segmentation network output binary masks for the heart and hilar regions. Attention bias injection, in self-attention computation, uses the anatomical mask as a bias term to enhance the model's focus on pathological regions: （2） in, This is a block-level mask matrix. If a block contains a pathological region, it is set to 1; otherwise, it is set to 0. These are learnable parameters; Medical semantic location coding is performed on chest X-ray images. Six anatomical landmarks are predefined as anatomical key points. Weighted coding is performed on the inverse distance. The distance between the center coordinates p=(x,y) of each image block and the anatomical key points is calculated to generate semantic location codes. （3） in, The normalized coordinates of the k-th key point are... The learnable weight matrix maps the distance to the feature space. The distance is Euclidean, which measures the proximity of an image patch to anatomical landmarks.

4. The chest pain classification method based on multimodal data fusion and deep learning model as described in claim 3, characterized in that, In step (22), the specific steps are as follows: For electrocardiogram (ECG) temporal feature extraction, a multi-scale temporal convolutional pyramid network is constructed. Feature extraction is performed from three aspects: short-time feature extraction, mid-time feature extraction, and long-time feature modeling. Short-time feature extraction uses a 5ms convolutional kernel to capture QRS wave details; mid-time feature extraction uses a 15ms convolutional kernel to analyze ST segment morphological trends; and long-time feature modeling uses a bidirectional LSTM network to capture heart rate variability features. Based on this, feature fusion is performed, concatenating the multi-scale features and then reducing the dimensionality through a 1×1 convolution. The formula is as follows: （4） in, We generate feature tensors for different time scales, then construct a clinical waveform constraint loss function to align the ST segments and constrain the ST segment features predicted by the model. The actual value as indicated by the doctor The error is: （5） in, For the feature segment ST of the i-th sample output by the model, To ensure the waveform classification consistency of the true ST segment features extracted from the waveform annotation matrix, KL divergence is used to ensure the waveform probability distribution output by the model. Distribution of annotations Consistency: （6） in, The model predicts the QRS / ST / T wave probabilities. The actual waveform distribution is labeled; Construct a total loss function to jointly optimize feature alignment and distribution consistency: （7） in, To balance the weights of the two losses, The loss value for segment ST. This represents the loss value between the image and the annotation.

5. The chest pain classification method based on multimodal data fusion and deep learning model as described in claim 4, characterized in that, Step (32) is as follows: The gating network dynamically adjusts expert weights based on real-time clinical data, and consists of two parts: data-driven and rule-driven. The first part is data-driven weight calculation, which involves inputting multimodal features. and vital signs Construct the gate function: （8） in, To dynamically calculate the weights, These are variable parameters, and then the weights are adjusted according to rules, imposing hard constraints when a specific clinical event is detected. （9） The gating function varies under different constraints. When ST elevation is greater than or equal to 2 mm and k=2, the gating constraint is 0.7; when the heart rate is greater than 120 and k=4, the gating constraint is 0.5; and in other cases, it is [missing value]. Then, the two weights are combined using a synthesis method: （10） in, The sigmoid function restricts the weights to the [0,1] interval. To create learnable parameters, the contributions of data and rules must be balanced. This is the weight matrix of the gated network; Step (33) is as follows: Multimodal feature fusion is performed by concatenating features from multiple expert models using residual connections. （11） （12） in, Let k be the feature extraction function for expert k. The residual projection matrix is... The final fused feature vector has a dimension of 1024.

6. The chest pain classification method based on multimodal data fusion and deep learning model as described in claim 5, characterized in that, In step (4), the specific process is as follows: Construct an input feature concatenation method to combine the 1024-dimensional features output in step (3). Feature vector, and the 768-dimensional vector encoded from the patient's electronic medical record text. To splice: （13） in, For a dimension reduction matrix, The concatenation method outputs as follows: ; Diagnostic decision generation uses causal attention decoding to generate diagnostic content and LLaMA-Med's Transformer decoder to generate the diagnostic logic chain. Attention computation incorporates medical knowledge constraints. （14） in, For clinical rule mask matrix, For sequence length, Dimensions for each attention head; Structured output generation: The model output contains three parallel headers: a disease probability header, a treatment suggestion header, and a differential diagnosis header. The disease probability header is as follows: （15） in, Four-category weighting; The suggested handling header is: （16） in, The 12 standard procedures are followed. Differential diagnosis of head is: （17） in, An embedding vector library containing 30 chest pain-related diseases; Generate a natural language decision chain based on the above steps: Generate report templates that align with clinical reasoning.

7. A chest pain classification system based on multimodal data fusion and deep learning models, applied to the chest pain classification method based on multimodal data fusion and deep learning models as described in claim 1, characterized in that, include: The preprocessing module is used to preprocess and enhance chest X-ray images and electrocardiogram multimodal data; Extraction module for anatomical-temporal dual-stream feature extraction; The fusion module is used for dynamic fusion of clinical knowledge-guided MOEs. The inference module is used for diagnostic inference driven by large models.

Images