Myocardial hypertrophy cause classification method and equipment based on multi-modal data fusion

Abstract

The invention relates to a myocardial hypertrophy cause classification method and device based on multi-modal data fusion, and the method comprises the following steps: obtaining multi-modal data of a to-be-classified object, the multi-modal data comprising a heart MRI image, an echocardiogram, an electrocardiogram and clinical text information; performing standardization preprocessing on the modal data, and extracting pathogenesis specificity features corresponding to the modal data; performing adaptive weighting and aggregation on the etiological specificity features corresponding to the modal data through a middle-term fusion strategy based on an attention mechanism to obtain fusion features; and performing end-to-end classification on the fusion features by using a deep learning model to obtain a myocardial hypertrophy cause classification result. Compared with the prior art, the method provided by the invention solves the problems of insufficient single-mode discrimination and difficult fusion of cross-mode isomerism, and has the advantages of high recognition accuracy, high efficiency, strong robustness and the like.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and device for classifying the etiology of myocardial hypertrophy based on multimodal data fusion. Background Technology

[0002] Myocardial hypertrophy refers to a pathological or physiological state characterized by an abnormal increase in the thickness of the ventricular wall. Its etiology is complex and diverse, mainly including primary cardiomyopathy (such as hypertrophic cardiomyopathy, HCM), secondary heart disease (such as hypertensive heart disease, HHD), physiological adaptations (such as athlete's heart), infiltrative diseases (such as cardiac amyloidosis and sarcoidosis), and storage diseases (such as Fabry disease and Danon's disease). Accurately identifying the etiology of myocardial hypertrophy is of significant clinical importance for developing treatment plans and assessing prognostic risks.

[0003] Traditional diagnosis of myocardial hypertrophy relies primarily on imaging examinations (echocardiography, cardiac MRI), electrocardiography, clinical history taking, and laboratory tests. However, because different etiologies of myocardial hypertrophy overlap in morphology, function, and clinical manifestations, a single examination method is often insufficient for a definitive diagnosis, requiring experienced specialists to synthesize multiple test results for judgment. This is not only time-consuming and labor-intensive, but also prone to misdiagnosis or missed diagnosis, especially in primary healthcare institutions facing a shortage of qualified personnel.

[0004] In recent years, with the rapid development of artificial intelligence technology, deep learning-based medical image analysis has made significant progress in the field of disease diagnosis. For example, patent CN116993700B discloses a scar myocardial infarction identification and classification method based on cine magnetic resonance imaging and deep learning. However, due to the characteristics of hypertrophic cardiomyopathy, existing intelligent diagnostic methods still have many limitations and are difficult to meet actual clinical needs. First, there is the problem of multimodal data heterogeneity. The data formats, sampling frequencies, and information densities of cardiac MRI, ultrasound, electrocardiogram, and clinical texts vary greatly. How to design a unified feature representation space to achieve effective fusion is a key challenge. Second, there is a class imbalance problem. The sample size of common causes (such as hypertrophic cardiomyopathy) is much larger than that of rare causes, causing the model to favor the majority class. Finally, existing multimodal combination methods mostly adopt shallow fusion strategies and fail to deeply explore the complementarity and synergistic effects between modalities. Simple parameter splicing or classification result voting cannot capture complex nonlinear relationships, resulting in limited gains from multimodal fusion and insignificant improvement in diagnostic performance. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method and device for classifying the etiology of myocardial hypertrophy based on multimodal data fusion, which is highly accurate, efficient and robust.

[0006] The objective of this invention can be achieved through the following technical solutions: A method for classifying the etiology of myocardial hypertrophy based on multimodal data fusion includes the following steps: Acquire multimodal data of the object to be classified, including cardiac MRI images, echocardiograms, electrocardiograms, and clinical text information; Each modality of data was standardized and preprocessed separately, and the etiology-specific features corresponding to each modality of data were extracted separately. The etiology-specific features corresponding to each modality data are adaptively weighted and aggregated using an attention-based mid-term fusion strategy to obtain fused features; A deep learning model is used to perform end-to-end classification of the fused features to obtain the etiological classification results of myocardial hypertrophy.

[0007] Furthermore, the cardiac MRI images include at least a cine sequence, a delayed gadolinium enhancement sequence, a T1 mapping sequence, and a T2 mapping sequence; The electrocardiogram is an image with a recording duration of ≥10 seconds and containing at least 5 complete cardiac cycles; The echocardiogram includes at least the parasternal left ventricular long-axis view, the apical four-chamber view, and the parasternal left ventricular short-axis view, and each view has images of two phases: end-diastole and end-systole. The clinical text information is extracted from the electronic medical record system.

[0008] Furthermore, the standardized preprocessing of the cardiac MRI images includes: For the movie sequence: select the end-diastolic frame as the reference frame, and uniformly adjust the pixels, time dimension, and grayscale of all Cine images in the movie sequence; For the delayed gadolinium enhancement sequence: extract the fiberized portion of the frame results and perform uniform enhancement on the data; For the T1 mapping sequence and the T2 mapping sequence: calculate the T1 value, extracellular volume fraction, and T2 value respectively, and generate a parameter map; The four sequences were registered to a unified coordinate system. Using the short-axis end-diastolic frame of the film sequence as a reference, the four sequences were aligned using an affine transformation based on the left ventricular centroid and the long axis direction.

[0009] Furthermore, the standardized preprocessing of the echocardiogram includes: Use adaptive median filtering to suppress speckle noise; Use the Laplacian operator to enhance edge features; The left ventricular region is automatically located and the region of interest is cropped using a target detection network.

[0010] Furthermore, the standardized preprocessing of the electrocardiogram includes: Detect lead detachment, baseline drift, and power frequency interference, and eliminate signals that do not meet quality standards; High-frequency noise is removed using a bandpass filter; The Pan-Tompkins algorithm was used to detect the location of the R-wave and to extract heartbeat segments centered on the R-wave. A standard timing matrix is ​​obtained through multi-lead synchronization and amplitude normalization.

[0011] Furthermore, the standardization preprocessing of the clinical text information includes: Data cleaning removes non-standard character sets and useless redundant data.

[0012] Furthermore, the extraction of etiology-specific features corresponding to each modality of data specifically involves: A three-dimensional convolutional neural network is used to extract the spatial-temporal joint representation of cardiac MRI images. A two-dimensional convolutional neural network is used to extract multi-view features from echocardiograms. A CNN-LSTM hybrid structure is used to extract temporal features from electrocardiograms. Natural language processing technology is used to extract semantic features from clinical text information.

[0013] Furthermore, the adaptive weighting and aggregation of the etiology-specific features corresponding to each modality data through an attention-based intermediate fusion strategy specifically includes: Each of the etiology-specific features is mapped to a unified dimension through an independent fully connected layer, and the mapping results are subjected to L2 normalization to obtain a standardized modal feature vector. The modal feature vectors are concatenated along the channel dimension and input into the attention network to calculate the attention weights of each modality. Initial fusion features are then generated according to the weighting rules. Bilinear pooling is introduced to capture second-order interaction information between modes. Bilinear features are obtained for each pairwise combination of modes and then concatenated with the initial fusion features to obtain the final fusion features.

[0014] Furthermore, the bilinear features are obtained by multiplying the normalized features of mode i by the transpose of the learnable weight matrix and then by the normalized features of mode j.

[0015] Furthermore, when calculating the attention weights for each modality, a modality-deficient adaptive mechanism is introduced, specifically as follows: When data for a certain modality is missing, the corresponding position weight is forcibly set to 0 during attention weight calculation, and the attention weights of the remaining available modalities are renormalized so that the total weight is 1.

[0016] The present invention also provides an electronic device including one or more processors, a memory, and one or more programs stored in the memory, said one or more programs including instructions for executing the myocardial hypertrophy etiology classification method based on multimodal data fusion as described above.

[0017] Compared with the prior art, the present invention has the following beneficial effects: I. High Classification and Recognition Rate: This invention integrates multi-source information such as MRI, ultrasound, ECG, and text to comprehensively analyze the morphological, functional, and electrophysiological characteristics of the heart. A mid-term fusion strategy based on an attention mechanism achieves feature-level fusion, overcoming the limitations of single-modal methods in etiological differentiation and significantly improving the classification accuracy and model robustness for complex etiologies. Furthermore, this invention performs standardized preprocessing on the initial collected data, enabling the extraction of more accurate etiology-specific features from each modality, further improving classification and recognition accuracy.

[0018] II. Automatic Classification: This invention can automatically identify the cause of myocardial hypertrophy without human intervention. It eliminates the need for doctors to manually extract features or compare image results one by one. Only multimodal data needs to be input to automatically complete feature extraction and classification, significantly reducing the workload of doctors.

[0019] III. Real-time Performance and Practicality: Under moderate computing resource configuration, the complete diagnostic process for a single case (including preprocessing, feature extraction, and fusion classification) can be completed within a clinically acceptable timeframe, meeting the needs of real-time clinical diagnosis. In particular, for complex cases, traditional diagnosis may require consultation and repeated discussions among multiple experts, while this invention can quickly provide diagnostic references.

[0020] IV. The model has strong generalization ability: Through comprehensive data preprocessing and standardized processes, this invention enables the model to have good transferability and adapt to different hospital equipment and scanning parameters.

[0021] V. Enhanced Interactivity: The method of this invention can be deployed in an electronic device, and the classification process can be controlled through a human-computer interaction interface. It is intelligent, fast, simple and clear, with operation instructions that are easy to understand. It supports functions such as data uploading, parameter adjustment, result visualization, and report export, making it convenient for clinicians to use.

[0022] VI. Clinical application value: This invention combines multimodal fusion technology with deep learning algorithms to significantly improve the accuracy and stability of etiological diagnosis of myocardial hypertrophy, providing reliable decision support for clinical practice, and has important clinical application prospects and industrialization value. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the training process of the deep learning model of the present invention; Figure 2 This is a schematic diagram of the multimodal feature extraction and fusion process of the present invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0025] This embodiment provides a method for classifying the etiology of hypertrophic myocardial infarction based on multimodal data fusion, including: acquiring multimodal data of the object to be classified, including cardiac MRI images, echocardiograms, electrocardiograms, and clinical text information; performing standardized preprocessing on each modality of data and extracting etiology-specific features corresponding to each modality of data; adaptively weighting and aggregating the etiology-specific features corresponding to each modality of data through an intermediate fusion strategy based on an attention mechanism to obtain fused features; and using a deep learning model to perform end-to-end classification on the fused features to obtain the etiology classification result of hypertrophic myocardial infarction.

[0026] The above method integrates multi-source information such as MRI, ultrasound, ECG and text, and performs standardized preprocessing on the initial data collected. The mid-term fusion strategy based on the attention mechanism achieves feature-level fusion, which overcomes the limitations of single-modal methods in etiological differentiation and significantly improves the classification accuracy of complex etiologies.

[0027] In this embodiment, the deep learning model adopts a three-layer fully connected classification network model, referencing... Figure 1 As shown, the specific training process of a deep learning model includes: Step S101: Acquire the patient's multimodal data, including cardiac MRI images, echocardiograms, electrocardiograms, and the patient's clinical text information. All data will be matched and time-synchronized using the patient's unique ID to create a case index.

[0028] The dataset used in this embodiment is derived from clinical cases, covering common causes and types of hypertrophic cardiomyopathy, distributed as follows: 120 cases of hypertrophic cardiomyopathy; 100 cases of hypertensive heart disease; 80 cases of athlete's heart; 120 cases of infiltrative cardiomyopathy, including 80 cases of cardiac amyloidosis and 60 cases of cardiac sarcoidosis; and 80 cases of storage cardiomyopathy, including 35 cases of Fabry disease, 25 cases of Danon's disease, and 20 cases of PRKAG2 syndrome. Each patient's data includes four modalities: cardiac MRI images, echocardiography, electrocardiography, and clinical text information.

[0029] During the data acquisition phase, basic requirements were set for each modality of data to ensure the reliability of subsequent classification and recognition. These basic requirements include: cardiac MRI images must include at least cine sequence, delayed gadolinium enhancement sequence, T1 mapping sequence, and T2 mapping sequence; electrocardiograms must be recorded for ≥10 seconds and contain at least 5 complete cardiac cycles; and echocardiograms must include at least the parasternal left ventricular long-axis view, the apical four-chamber view, and the parasternal left ventricular short-axis view, with each section containing images of both end-diastolic and end-systolic phases.

[0030] Specifically, this embodiment uses a Siemens Magnetom Skyra 3.0T MRI scanner for cardiac MRI scans. The scanning sequences include cine sequences, delayed gadolinium enhancement sequences (LGE sequences), T1 mapping sequences, and T2 mapping sequences. The cine sequence uses a steady-state free precession sequence, covering the short-axis, long-axis, and four-chamber views of the heart, acquiring 25 to 30 frames per cardiac cycle. The delayed gadolinium enhancement sequence is acquired 10 to 15 minutes after intravenous injection of gadolinium contrast agent, using a reverse recovery fast gradient echo sequence, with the inversion time optimized based on the disappearance of normal myocardial signal. The T1 mapping sequence uses a modified Look-Locker reverse recovery sequence, acquired before and after contrast agent injection, generating raw T1 maps and post-enhanced T1 maps, and calculating the extracellular volume integral. The T2 mapping sequence uses a multi-echo spin echo sequence to generate T2 parameter maps. The MRI data generated for each patient includes approximately 200 to 300 DICOM format image files.

[0031] Electrocardiogram (ECG) data acquisition was performed using a standard 12-lead ECG machine. The sampling rate for recording ECG signals was set to 1000 Hz, and the recording duration for each patient was 10 seconds, ensuring that at least 8 to 10 complete cardiac cycles were included. ECG leads included limb leads and chest leads. Data was stored in XML format, containing raw signal waveforms and automatically measured parameters.

[0032] Echocardiographic data were acquired using a Philips color Doppler ultrasound diagnostic system. Standard ultrasound views included the parasternal left ventricular long-axis view, the apical four-chamber view, and the parasternal left ventricular short-axis view, with one static image acquired at end-diastole and end-systole for each view.

[0033] Clinical text information is extracted from the hospital's electronic medical record system and includes a written description of the patient's chief complaint, present illness, past medical history, family history, physical examination, laboratory test results, and previous imaging reports. The chief complaint typically records the patient's main symptoms and their duration, such as chest tightness, shortness of breath, or cardiac enlargement found during physical examination. The present illness details the onset, development, and accompanying symptoms. The past medical history focuses on relevant medical histories such as hypertension, diabetes, and coronary heart disease. The family history section pays particular attention to family histories of cardiomyopathy or sudden cardiac death. The physical examination records signs such as murmurs and arrhythmias found during cardiac auscultation. Laboratory tests include cardiac enzyme profiles, brain natriuretic peptide, troponin, and genetic testing results. Previous imaging reports include the findings of previous echocardiograms, CT scans, and MRI scans. Clinical text is stored in a mixed structured and unstructured format, with an average text length of approximately 500 to 1000 words per patient. To achieve accurate matching and synchronous access to data across different modalities, this embodiment assigns a unique case number to each patient, formatted as a disease type prefix followed by a serial number, and establishes a data index table to record the file paths, acquisition times, and data integrity markers for the four modalities of data for each patient.

[0034] Step S102: Perform standardization preprocessing on the multimodal data.

[0035] To improve training efficiency and inference timeliness, reduce computational burden, and eliminate invalid background to the greatest extent possible while ensuring the integrity of key pathological and physiological information, this embodiment uniformly designs a standardized preprocessing process that is divided into modes and stages before model input parameters.

[0036] Specifically, the electrocardiogram (ECG) data first undergoes quality checks to identify and remove unqualified ECG signals with severe lead dropouts, baseline drift exceeding 0.5 mV, or power line interference amplitude exceeding 20% ​​of the signal amplitude. For qualified signals, a 3-40 Hz bandpass Butterworth filter is used to filter out high-frequency electromyographic noise and low-frequency baseline drift. Subsequently, the Pan-Tompkins algorithm is used for QRS complex detection, extracting heartbeat segments containing complete P-QRS-T complexes centered on the detected R wave position. Time alignment of the 12-lead signals ensures consistent R wave positions across all leads. Z-score normalization is used to normalize the amplitude of each lead to eliminate individual differences, ultimately generating a standard time-series matrix of size 12 × 600.

[0037] Echocardiographic data preprocessing employed adaptive median filtering for noise reduction, addressing the speckle noise characteristics of ultrasound images. The filter window size was adaptively adjusted from 3×3 to 7×7 pixels based on local image statistical characteristics. To enhance the boundary features between the myocardium and cardiac chambers, the Laplacian operator was used for edge enhancement, with an 8-neighborhood template. A pre-trained YOLOv5 object detection network was used to automatically locate the left ventricular region. This network, trained on a large-scale ultrasound image dataset, accurately detected the left ventricular bounding box. The region of interest (ROI) was cropped based on the detection results and uniformly adjusted to 256×256 pixels using bilinear interpolation. Images from the end-diastolic and end-systolic phases were retained for each of the three standard sections, resulting in six preprocessed ultrasound images. Gray-level normalization was then performed to map pixel values ​​between 0 and 1.

[0038] The preprocessing of multiple cardiac MRI sequences is quite complex, including: for cine sequences, end-diastolic frames are selected as reference frames, and all cine images in the cine sequence are uniformly adjusted in terms of pixels, time dimension, and grayscale; for delayed gadolinium enhancement sequences, the fibrotic portion of the frame results are extracted, and the data are uniformly enhanced; for T1 mapping sequences and T2 mapping sequences, T1 values, extracellular volume integrals, and T2 values ​​are calculated respectively to generate parametric maps; finally, the four sequences are registered to a unified coordinate system.

[0039] Specifically, for the Cine sequence, the left ventricular cavity area change curve of each slice over time was analyzed, and the moment with the largest left ventricular cavity area was selected as the end-diastolic phase. All Cine images were adjusted to a uniform 256×256 pixel spatial resolution using bicubic interpolation. In the time dimension, if the original number of frames was less than 30, cubic spline interpolation was used for upsampling; if it was more than 30 frames, equal-interval downsampling was used to unify it to 30 frames. Z-score normalization was applied to the grayscale value of each pixel to make the mean of the entire sequence 0 and the standard deviation 1, eliminating signal intensity differences between different scans. For the LGE sequence, due to its characteristic of displaying myocardial fibrosis and scar areas, high-signal fibrotic regions were segmented based on a signal intensity threshold, which was usually set to the normal myocardial signal intensity plus 5 standard deviations. Histogram equalization was applied to the images to maximize the contrast between fibrotic regions and normal myocardium. The processed LGE images were spatially normalized to 256×256 pixels, and the grayscale values ​​were normalized to between 0 and 1.

[0040] T1 and T2 mapping are processed by calculating T1 and T2 values ​​using pixel-level nonlinear fitting from multiple images acquired at different inversion or echo times. The extracellular volume integral is calculated as 1 minus the ratio of myocardial T1 change rate to blood pool T1 change rate. When generating pseudo-color parameter maps, Jet color maps are used to map parameter values ​​to colors for easier visualization.

[0041] Multiple sequence registration is a crucial step in MRI preprocessing. The LGE, T1 mapping, and T2 mapping sequences were registered to the coordinate system of the short-axis end-diastolic frame of the Cine sequence. First, the spatial location information in the DICOM file header of each sequence was read, and this metadata was used to transform each sequence to a unified patient anatomical coordinate system, achieving initial spatial alignment. Then, affine transformation matrices were used for registration, with transformation parameters including translation, rotation, and scaling. Mutual information was used as a similarity metric, and the Powell optimization algorithm was employed to solve for the optimal transformation parameters.

[0042] Clinical text data preprocessing: The raw text exported from the electronic medical record system contains a large number of non-standard characters, special symbols, and redundant spaces. First, data cleaning is performed to remove HTML tags, special symbols, and redundant whitespace, standardize Chinese punctuation, and correct obvious typos. A pre-trained Chinese medical BERT model is used for named entity recognition. This model, pre-trained on a large-scale Chinese medical corpus, can recognize medical entities such as diseases, symptoms, examinations, drugs, and genes. Specific entity types identified include disease entities such as hypertension, diabetes, and atrial fibrillation; symptom entities such as chest tightness, shortness of breath, palpitations, and syncope; family history entities such as a father's sudden death or a brother's cardiomyopathy; gene mutations such as MYH7 gene mutations or specific gene locus variations; and examination indicators such as left ventricular wall thickness and brain natriuretic peptide levels. The BERT model is used to convert the cleaned text into a 768-dimensional semantic feature vector. The extracted entity information is encoded into structured features, including 20 structured features such as family history, gene mutation, maximum left ventricular wall thickness, and brain natriuretic peptide levels. The tendency score for each etiology is calculated based on the domain knowledge rules. For example, if there is a myosarcomin gene mutation and a family history, the tendency score for hypertrophic cardiomyopathy is higher. If there is a long history of hypertension and left ventricular concentric hypertrophy, the tendency score for hypertensive heart disease is higher.

[0043] Step S103: Divide the preprocessed multimodal data into a training set, a validation set, and a test set.

[0044] Specifically, during the dataset partitioning phase, the 500 patient data points were randomly divided into training, validation, and test sets in an 8:1:1 ratio to ensure that the proportions of the five etiologies in each dataset remained consistent with the overall dataset. The training set contained 400 cases: 96 with hypertrophic cardiomyopathy, 80 with hypertensive heart disease, 64 with athlete's heart disease, 96 with infiltrative cardiomyopathy, and 64 with storage cardiomyopathy. The validation and test sets each contained 50 cases, with the distribution of each etiology maintaining a similar proportion to the training set. A stratified random sampling method was used for partitioning to ensure a balanced distribution of each subclass across the three datasets. A random seed of 42 was set to ensure the reproducibility of the partitioning results.

[0045] Step S104: Extract etiology-specific features from each modality of data.

[0046] Specifically, this embodiment designs a specialized deep learning network architecture tailored to the characteristics of different modalities of data. (Reference) Figure 2 As shown, a three-dimensional convolutional neural network is used to extract the spatial-temporal joint representation of cardiac MRI images. The network architecture is based on 3D ResNet-50 and optimized for multi-sequence cardiac MRI. To input sequence data of different dimensions into a unified 3D convolutional network, this embodiment selects the middle 5 slices covering the maximum diameter of the left ventricle as representative slices. To ensure that the LGE, T1 mapping, and T2 mapping sequences are consistent with the Cine sequence in the input tensor dimension, the three-dimensional data of these three sequences are copied and expanded 30 times in the temporal dimension to obtain a four-dimensional tensor of 256×256×5×30. Since the information of LGE, T1 mapping, and T2 mapping is static and does not change with the cardiac cycle, the copying operation in the temporal dimension does not introduce new information and is only used for dimension matching to facilitate unified processing by the network. The input tensors of the four sequences are batch size × 1 × 5 × 30 × 256 × 256, where 1 represents a single-channel grayscale image, 5 is the number of slices, and 30 is the number of time frames.

[0047] Independent 3D convolutional branches were set for the four sequences, each using the same network structure. Features extracted from the four branches were fused using an attention mechanism. The attention weight calculation module consisted of a fully connected layer with an input dimension of 512×4 (2048) and an output dimension of 256. After ReLU activation, another fully connected layer with an input dimension of 256 and an output dimension of 4 was applied. Finally, softmax normalization was used to obtain the attention weights for the four sequences. The fused feature was calculated as a weighted sum of the features of each sequence and their corresponding attention weights. After global average pooling, the slice, temporal, and spatial dimensions were compressed to obtain a 512-dimensional MRI feature vector. This feature vector contained information on cardiac morphology, function, tissue characteristics, and fibrosis.

[0048] Ultrasound image feature extraction employed a two-dimensional convolutional neural network, with the EfficientNet-B3 architecture as the base network. This architecture balances network depth, width, and resolution through a composite scaling method. Independent EfficientNet-B3 branches were set up for three standard views: parasternal left ventricular long-axis view, apical four-chamber view, and parasternal left ventricular short-axis view. The input to each branch was a 2×256×256 input tensor formed by concatenating two images of that view at end-diastole and end-systole. Each EfficientNet-B3 branch contained a Stem layer, seven MBConv blocks, and a Head layer. The Stem layer was a 3×3 convolution with a stride of 2 and 40 output channels. The MBConv blocks used a moving inverse residual bottleneck structure and a Squeeze-and-Excitation attention mechanism to progressively extract multi-scale features. The Head layer was a 1×1 convolution with 1536 output channels. The difference feature between end-diastole and end-systole features was calculated; this difference feature captures the motion and deformation information of the heart from diastole to systole, reflecting the heart's systolic function. A multi-head attention mechanism was used to fuse features from three views, with four attention heads configured. Each head independently computed the Query, Key, and Value matrices. The attention weights were calculated by multiplying the matrix product of the transpose of the Query and Key, dividing by the square root of the Key vector dimension, normalizing using Softmax, and then multiplying by the Value matrix. The outputs of the four attention heads were concatenated and linearly transformed to obtain the fused ultrasound features. After global average pooling, a 384-dimensional ultrasound feature vector was obtained, which contained morphological information of the left ventricle, wall motion information, and systolic function indicators.

[0049] ECG feature extraction employs a CNN-LSTM hybrid structure to hierarchically represent the 12-lead ECG time-series signal. This structure combines the local feature extraction capabilities of CNNs with the long-range temporal dependency modeling capabilities of LSTMs. First, a one-dimensional convolutional layer is independently applied to each lead for local feature extraction. Then, the feature sequences extracted by the CNN are input into a bidirectional LSTM network. Simultaneously, an attention mechanism is used to weightedly sum the hidden states at each time step of the LSTM to obtain global context features. The attention weights are calculated by a small neural network, with the input being the hidden state at each time step and the output being the corresponding attention score normalized by Softmax. The final ECG feature vector is a concatenation of the forward hidden state, backward hidden state, and global context features, with a dimension of 256. This feature vector includes morphological features of the ECG signal, such as QRS wave width and amplitude, rhythm features such as heart rate variability, and spatial relationships between leads.

[0050] Clinical text feature extraction employed natural language processing (NLP) techniques, with the pre-trained model using the Transformer-based BERT model, specifically the Chinese medical BERT model. The text encoding process began with a WordPiece segmenter to divide the text into sub-word units and insert special markers. These sub-words were then mapped to 768-dimensional vectors with positional encoding. A 12-layer Transformer encoder, each layer incorporating multi-head self-attention and a feedforward neural network, was used, with a total of approximately 110 million parameters. The output vector corresponding to the special markers was extracted as the semantic representation of the entire text, with a dimension of 768. The 20 structured features and 5 etiology propensity scores extracted in the preprocessing were encoded into a 64-dimensional feature vector through a fully connected layer. This fully connected layer used the ReLU activation function and Dropout for regularization. The 768-dimensional BERT semantic features and the 64-dimensional structured features were concatenated to form an 832-dimensional comprehensive text feature vector, which was then further reduced in dimensionality by a fully connected layer to obtain the final 256-dimensional text features. This feature vector integrates the semantic information of the clinical text with structured medical indicators, comprehensively reflecting the patient's clinical manifestations and auxiliary examination results.

[0051] Step S105: Adaptively weight and aggregate the four modal features using an attention-based mid-term fusion strategy.

[0052] The feature vectors extracted from the four modalities have inconsistent dimensions: MRI (512-dimensional), ultrasound (384-dimensional), ECG (256-dimensional), and clinical text (256-dimensional). To achieve effective fusion, all modal features need to be mapped to a unified dimension. An independent fully connected layer is set for each modality: MRI is mapped using a fully connected layer from 512 to 256, ultrasound using a fully connected layer from 384 to 256, and ECG and clinical text using a fully connected layer from 256 to 256. L2 normalization is then performed on the mapped feature vectors to make the feature vector magnitude equal to 1, eliminating scale differences between the different modalities.

[0053] The four standardized modal feature vectors are concatenated along the channel dimension to a dimension of 1024. The concatenated features are then input into an attention network to calculate the attention weights for each modality. The attention network consists of a first fully connected layer (1024 to 512 dimensions) activated by ReLU, a second fully connected layer (512 to 128 dimensions) activated by ReLU, and an output layer (128 to 4 dimensions) normalized using Softmax. The output attention weights are four-dimensional vectors whose components sum to 1 and are all non-negative. A fused feature vector is generated according to a weighting rule, which is the weighted sum of the normalized features of each modality and their corresponding attention weights, with a dimension of 256. This fused feature vector is then increased to 512 dimensions using a fully connected layer and activated by ReLU, yielding the initial fused features. (Refer to...) Figure 2 As shown.

[0054] In practical clinical applications, some patients may be unable to complete certain examinations for various reasons, resulting in missing data for a particular modality. This invention designs a modality missing adaptive mechanism that enables the system to make accurate diagnoses even when modalities are incomplete. The modality missing detection mechanism calculates the L2 norm of the feature vector for each modality. When the L2 norm is lower than a preset threshold of 0.01, the modality is determined to be missing. The feature vector of the missing modality is set to a zero vector or filled with a predefined missing marker. During attention weight calculation, the weights at the corresponding positions of the detected missing modality are forcibly set to 0. The attention weights of the remaining available modalities are renormalized so that the sum of the weights is still 1. The fused features are recalculated based on the renormalized weights, ensuring that the system can still make diagnoses using available modality information even when modalities are missing.

[0055] Bilinear pooling is introduced to capture second-order interaction information between modalities. Bilinear pooling can model the multiplicative interaction between features of two modalities, revealing synergistic effects between modalities. Bilinear features are calculated for each of the six pairs of any two modalities. The formula for calculating bilinear features is: the transpose of the normalized feature of modality i multiplied by the learnable weight matrix, and then multiplied by the normalized feature of modality j. The weight matrix is ​​a 256×256 learnable parameter matrix obtained through backpropagation. The bilinear features are scalars representing the interaction strength between modalities i and j.

[0056] Six bilinear features are concatenated into a 6-dimensional vector, which is then subjected to nonlinear transformation and dimensionality increase through a fully connected layer to obtain an interaction feature vector with a dimension of 64. The 512-dimensional fusion feature vector after dimensionality increase is concatenated with the 64-dimensional interaction feature vector, resulting in a dimension of 576. Dimensionality reduction and feature integration are then performed through a fully connected layer to obtain the final enhanced fusion feature vector with a dimension of 256, which serves as the final fusion feature. This enhanced fusion feature vector contains complementary information from the four modalities, second-order interaction features between modalities, and multi-level abstract semantic information, which serves as input to the subsequent classification network.

[0057] Step S106: Use a deep learning model to perform end-to-end etiological classification on the fused features and output the classification results.

[0058] Based on the fusion features of data from the training, validation, and test sets, and the known etiological categories, a deep learning model is trained. When classifying the object to be classified, based on steps S101, S102, S104, and S105, the fusion features corresponding to the object are obtained. The trained deep learning model is then used to obtain the etiological category result corresponding to the object, serving as auxiliary reference information for doctors' etiological diagnosis.

[0059] Specifically, the 256-dimensional enhanced fusion features are input into a three-layer fully connected classification network. The first layer is a fully connected layer with an input dimension of 256 and an output dimension of 128, followed by a ReLU activation function and a batch normalization layer, with a dropout rate of 0.4. The second layer is also a fully connected layer with an input dimension of 128 and an output dimension of 64, followed by a ReLU activation function and a batch normalization layer, with a dropout rate of 0.3. The output layer is a fully connected layer with an input dimension of 64 and an output dimension of 5, corresponding to five etiologies: hypertrophic cardiomyopathy, hypertensive heart disease, athlete's heart, infiltrative cardiomyopathy, and metabolic / storage cardiomyopathy.

[0060] The five neurons in the output layer output unnormalized logits values. The logits are converted into a probability distribution using the Softmax function, and the predicted class is the one with the highest probability. Focal Loss is used as the main loss function to address the class imbalance problem in the dataset. The model is trained using the AdamW optimizer, an improved version of the Adam optimizer that decouples weight decay from gradient updates, providing better regularization. Cosine annealing is used as the learning rate scheduling strategy, and the model is trained using data from 400 patients in the training set. In each training epoch, the training set is randomly shuffled and divided into 25 batches of size 16. After each training epoch, the model performance is evaluated on a validation set of 50 cases, recording metrics such as accuracy, precision, recall, and F1 score. The learning rate is adjusted based on the validation set performance to preserve the optimal model parameters. Through this training process, the model can learn the specific feature patterns of each etiology. Validation results on a dataset containing 500 patients show that the method of this invention can accurately distinguish five types of myocardial hypertrophy etiologies, significantly outperforming existing single-modality methods and simple fusion strategies.

[0061] If the above methods are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0062] In another embodiment, an electronic device may also be provided, including one or more processors, a memory, and one or more programs stored in the memory, the one or more programs including instructions for performing the myocardial hypertrophy etiology classification method based on multimodal data fusion as described above.

[0063] Specifically, the electronic device provides a human-computer interaction interface, which can control the classification process. It is intelligent, fast, simple and clear, with operation instructions that are easy to understand. It supports functions such as data uploading, parameter adjustment, result visualization, and report export, making it convenient for clinicians to use.

[0064] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0067] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for classifying the etiology of hypertrophic cardiomyopathy based on multimodal data fusion, characterized in that, Includes the following steps: Acquire multimodal data of the object to be classified, including cardiac MRI images, echocardiograms, electrocardiograms, and clinical text information; Each modality of data was standardized and preprocessed separately, and the etiology-specific features corresponding to each modality of data were extracted separately. The etiology-specific features corresponding to each modality data are adaptively weighted and aggregated using an attention-based mid-term fusion strategy to obtain fused features; A deep learning model is used to perform end-to-end classification of the fused features to obtain the etiological classification results of myocardial hypertrophy.

2. The method for classifying the etiology of myocardial hypertrophy based on multimodal data fusion according to claim 1, characterized in that, The cardiac MRI images include at least cinema sequences, delayed gadolinium enhancement sequences, T1 mapping sequences, and T2 mapping sequences; The electrocardiogram is an image with a recording duration of ≥10 seconds and containing at least 5 complete cardiac cycles; The echocardiogram includes at least the parasternal left ventricular long-axis view, the apical four-chamber view, and the parasternal left ventricular short-axis view, and each view has images of two phases: end-diastole and end-systole. The clinical text information is extracted from the electronic medical record system.

3. The method for classifying the etiology of myocardial hypertrophy based on multimodal data fusion according to claim 2, characterized in that, Standardized preprocessing of the cardiac MRI images includes: For the movie sequence: select the end-diastolic frame as the reference frame, and uniformly adjust the pixels, time dimension, and grayscale of all Cine images in the movie sequence; For the delayed gadolinium enhancement sequence: extract the fiberized portion of the frame results and perform uniform enhancement on the data; For the T1 mapping sequence and the T2 mapping sequence: calculate the T1 value, extracellular volume fraction, and T2 value respectively, and generate a parameter map; The four sequences were registered to a unified coordinate system. Using the short-axis end-diastolic frame of the film sequence as a reference, the four sequences were aligned using an affine transformation based on the left ventricular centroid and the long axis direction.

4. The method for classifying the etiology of myocardial hypertrophy based on multimodal data fusion according to claim 1 or 2, characterized in that, The standardized preprocessing of the echocardiogram includes: Use adaptive median filtering to suppress speckle noise; Use the Laplacian operator to enhance edge features; The left ventricular region is automatically located and the region of interest is cropped using a target detection network.

5. The method for classifying the etiology of myocardial hypertrophy based on multimodal data fusion according to claim 1 or 2, characterized in that, The standardized preprocessing of the electrocardiogram includes: Detect lead detachment, baseline drift, and power frequency interference, and eliminate signals that do not meet quality standards; High-frequency noise is removed using a bandpass filter; The Pan-Tompkins algorithm was used to detect the location of the R-wave and to extract heartbeat segments centered on the R-wave. A standard timing matrix is ​​obtained through multi-lead synchronization and amplitude normalization.

6. The method for classifying the etiology of myocardial hypertrophy based on multimodal data fusion according to claim 1, characterized in that, The specific steps for extracting the etiology-specific features corresponding to each modality of data are as follows: A three-dimensional convolutional neural network is used to extract the spatial-temporal joint representation of cardiac MRI images. A two-dimensional convolutional neural network is used to extract multi-view features from echocardiograms. A CNN-LSTM hybrid structure is used to extract temporal features from electrocardiograms. Natural language processing technology is used to extract semantic features from clinical text information.

7. The method for classifying the etiology of myocardial hypertrophy based on multimodal data fusion according to claim 1, characterized in that, The adaptive weighting and aggregation of the etiology-specific features corresponding to each modality data through an attention-based intermediate fusion strategy specifically includes: Each of the etiology-specific features is mapped to a unified dimension through an independent fully connected layer, and the mapping results are subjected to L2 normalization to obtain a standardized modal feature vector. The modal feature vectors are concatenated along the channel dimension and input into the attention network to calculate the attention weights of each modality. Initial fusion features are then generated according to the weighting rules. Bilinear pooling is introduced to capture second-order interaction information between modes. Bilinear features are obtained for each pairwise combination of modes and then concatenated with the initial fusion features to obtain the final fusion features.

8. The method for classifying the etiology of myocardial hypertrophy based on multimodal data fusion according to claim 7, characterized in that, The bilinear features are obtained by multiplying the normalized feature transpose of mode i by the learnable weight matrix and then by the normalized feature of mode j.

9. The method for classifying the etiology of myocardial hypertrophy based on multimodal data fusion according to claim 7, characterized in that, When calculating the attention weights for each modality, a modality-deficient adaptive mechanism is introduced, specifically: When data for a certain modality is missing, the corresponding position weight is forcibly set to 0 during attention weight calculation, and the attention weights of the remaining available modalities are renormalized so that the total weight is 1.

10. An electronic device, characterized in that, It includes one or more processors, a memory, and one or more programs stored in the memory, said one or more programs including instructions for executing the multimodal data fusion-based etiology classification method for myocardial hypertrophy as described in any one of claims 1-9.

Images