Multi-modal data fusion method, MASLD data processing method, system, equipment and medium

Abstract

The invention discloses a multi-modal data fusion method, an MASLD data processing method, an MASLD data processing system, MASLD data processing equipment and a medium, and belongs to the technical field of artificial intelligence and medical information crossing. According to the method, a tongue picture image and structured health examination data are fused, a convolutional neural network and a graph attention network are used for extracting visual features and medical features respectively, feature fusion is carried out through an attention mechanism, and non-invasive and high-precision prediction of the metabolism-related fatty liver disease (MASLD) is achieved. The problems that in the prior art, early diagnosis of MASLD is difficult and depends on invasive examination are solved, a novel and extensible solution is provided for non-invasive MASLD screening, and early risk stratification and clinical decision are expected to be enhanced.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and medical information technology, and specifically relates to a multimodal data fusion method and a MASLD data processing method, system, device and medium. Background Technology

[0002] Metabolic fatty liver disease (MASLD) is often asymptomatic in its early stages, and traditional biochemical tests are often insufficient to detect hepatic steatosis or inflammation. This limitation can lead to missed diagnoses in a large number of high-risk individuals. Standard diagnostic methods typically integrate clinical history, laboratory parameters, and imaging studies. While liver biopsy remains the gold standard for definitive diagnosis, providing precise histological insights into steatosis, inflammation, and fibrosis, its invasiveness, cost, and associated risks make it unsuitable for large-scale population screening.

[0003] To address these limitations, recent advances in artificial intelligence have spurred increasing interest in using non-invasive biomarkers and machine learning algorithms for MASLD risk prediction. Early research in this field has yielded encouraging results. However, many existing models have significant limitations: some are designed for specific populations, some heavily rely on high-dimensional omics data or specific biomarkers, and many others are limited by small sample sizes or insufficient consideration of population heterogeneity.

[0004] Therefore, there is still an urgent need for a robust, scalable, and universally applicable MASLD risk prediction tool to be suitable for large-scale health screening of different populations. Summary of the Invention

[0005] This invention aims to address the problems existing in the prior art by providing a multimodal data fusion method and a data processing method, system, device, and medium for metabolic-associated fatty liver disease (MASLD) based on multimodal data fusion. By fusing tongue images and structured health examination data, it utilizes deep learning technology to achieve high-precision MASLD risk prediction and classification.

[0006] This invention provides a multimodal data fusion method, comprising: Obtain a multimodal dataset, which includes structured data of a first modality and image data of a second modality; A data relationship graph structure is constructed based on the structured data; The data relationship graph structure is processed using a graph neural network to extract a first feature representation; The image data is processed using a convolutional neural network to extract a second feature representation; Multimodal fusion features are obtained by fusing the first feature representation and the second feature representation through an attention fusion mechanism; Based on the aforementioned multimodal fusion features, the data processing results are output.

[0007] Preferably, the construction of the data relationship graph structure based on structured data includes: Calculate the similarity between different data samples; The connection relationships between data samples are determined based on similarity thresholds or the k-nearest neighbor algorithm; Construct a graph structure G=(V, E), where V represents the set of nodes, each node corresponds to a data sample, and E represents the set of edges, with edge weights calculated based on similarity. And / or, the attention fusion mechanism includes: The first and second feature representations are projected onto the same feature space; preferably, the graph neural network and the convolutional neural network are embedded and independently projected onto the same shared feature space through linear transformation and LayerNorm. Calculate the attention weights for each modal feature; preferably, using the following formula: Attention weights are computed to adaptively fuse modalities, where The activation function is softmax, and W1 and W2 are attention weights. , For feature projection; The modal features are weighted and fused according to the attention weights to obtain multimodal fusion features; preferably, a multilayer perceptron classifier is used to fuse the features and then predict the class label. Furthermore, the multilayer perceptron classifier is preferably a classifier composed of three fully connected layers. And / or, after fusion, the method further includes: combining domain knowledge to interpret and verify the feature importance, and generating feature importance output results.

[0008] Preferably, the method further includes a training process, employing a two-stage training strategy: The first stage involves training the graph neural network module separately; preferably, in the first stage, a data relationship graph is constructed based on structured data, and the graph neural network is used to process the data relationship graph, extract feature representations, and train a prediction task based on the extracted features. In the second stage, the graph neural network parameters are fixed, and the feature fusion module and output module are trained. Preferably, in the second stage, a pre-trained graph neural network is used to extract structured feature representations, and a convolutional neural network is used to extract image feature representations. Finally, multimodal features are fused through an attention fusion mechanism, and prediction task training is performed based on the fused multimodal features.

[0009] The present invention also provides a multimodal data fusion system, comprising: The data acquisition module is configured to acquire a multimodal dataset, which includes structured data of a first modality and image data of a second modality. The graph building module is configured to build data relationship graphs based on structured data. The graph neural network module is configured to use a graph neural network to process the data relationship graph structure and extract a first feature representation. A convolutional neural network module is configured to process the image data using a convolutional neural network and extract a second feature representation; The fusion module is configured to fuse the first feature representation and the second feature representation through an attention fusion mechanism to obtain multimodal fusion features; The output module is configured to output the data processing results based on the multimodal fusion features.

[0010] This invention also provides a MASLD data processing method based on multimodal data fusion, comprising the following steps: Obtain a multimodal dataset related to MASLD, which includes structured clinical data and tongue image data; A patient similarity graph structure was constructed based on the MASLD-specific clinical characteristics in the aforementioned clinical data; The graph structure is processed using a graph neural network to extract clinical feature representations; The tongue image data is processed using a convolutional neural network to extract tongue image feature representations; Multimodal fusion features are obtained by fusing the clinical feature representations and tongue image feature representations through an attention fusion mechanism; Based on the aforementioned multimodal fusion features, the MASLD data processing results are output.

[0011] Preferably, the patient similarity map structure constructed based on MASLD-specific clinical features includes: Select a subset of clinical features associated with MASLD; Weights are assigned to different features based on their importance; The similarity between patients was calculated using a weighted similarity calculation method; Graph structures are constructed based on similarity thresholds or the k-nearest neighbor algorithm; And / or, the attention fusion mechanism includes: Project clinical feature representations and visual feature representations onto the same feature space; Calculate the attention weights for each modality feature, with the initial weights for clinical features being 0.5-0.7 and the initial weights for visual features being 0.3-0.5. Multimodal fusion features are obtained by weighting and fusing modal features based on attention weights.

[0012] Preferably, the graph neural network is a graph attention network, comprising a two-layer structure, wherein the first layer uses 6-10 attention heads and the second layer uses 1 attention head, and batch normalization and dropout techniques are used between layers; And / or, the convolutional neural network is a pre-trained ResNet-50 model, fine-tuned for the characteristics of tongue images, focusing on extracting tongue color, tongue coating thickness and tongue shape features; And / or, the structured clinical data includes at least one of liver function indicators, blood lipid indicators, blood glucose metabolism indicators, and anthropometric indicators.

[0013] This invention also provides a MASLD processing system based on multimodal data fusion, comprising: The data acquisition module is configured to acquire a MASLD-related multimodal dataset, which includes structured clinical data and tongue image data; The graph construction module is configured to construct a patient similarity graph structure based on the MASLD-specific clinical features in the clinical data. The feature extraction module is configured to use a graph neural network to extract clinical features from the graph structure and to use a convolutional neural network to extract tongue features from the tongue image data. The fusion module is configured to fuse the clinical feature representation and tongue image feature representation through an attention fusion mechanism to obtain multimodal fusion features; The output module is configured to output MASLD data processing results based on the multimodal fusion features.

[0014] This invention also provides a multimodal data processing system based on a B / S architecture, comprising: On the browser side, the system uploads the dataset required for input locally, which includes first modality data and second modality data. The server is used to run a preset multimodal fusion model to complete data processing and return the fusion processing results to the browser. The multimodal fusion model includes: a graph construction component for constructing a similarity structure graph based on first modality data; a dual-branch feature extraction component, including a graph neural network branch and a convolutional neural network branch, where the graph neural network processes the similarity graph structure to extract a first feature representation, and the convolutional neural network processes the second modality data to extract a second feature representation; a multimodal attention fusion component for adaptively fusing the first and second feature representations through an attention fusion mechanism to obtain multimodal fusion features; and a feature importance analysis and output component for evaluating the contribution of each multimodal fusion feature.

[0015] Compared with the prior art, the present invention has at least one or more of the following beneficial effects: This invention utilizes multimodal data fusion and processing technology for MASLD to fully leverage MASLD-related clinical indicators and tongue features, thereby improving the targeting of data processing; by constructing MASLD-specific maps, it better captures the similarity relationships between patients; by fusing multimodal information, it improves the accuracy and reliability of data processing results; and by providing interpretable data processing results, it contributes to medical research and clinical decision support.

[0016] This invention proposes a multimodal deep learning framework (DeepMASLD), which integrates tongue images with structured health data to overcome the limitations of insufficient information from single-modal data. Specifically, it utilizes CNN to extract color and texture features from the tongue image, GNN to dynamically model the correlations between clinical indicators, captures multi-scale spatial dependencies, and fuses multimodal features through an attention mechanism, thereby improving the accuracy and generalization of MASLD risk prediction. This makes it particularly suitable for resource-constrained primary healthcare settings.

[0017] Therefore, multimodal fusion-based processing is suitable for large-scale data processing, has good generalization ability, and can make full use of the complementary information between multimodal data, achieving an accuracy of over 92%. Moreover, it can provide interpretable processing results for MASLD, which is helpful for further analysis. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall framework of the DeepMASLD model of this invention; Figure 2 This is a performance comparison chart of various models in the MASLD risk classification task; Figures 3A-3B This is a distribution chart of the MASLD dataset and the distribution of CAP values ​​in different categories; Figures 4A-4D This is a visual evaluation graph of the multimodal model's performance on the MASLD test set; Figure 5 This is a performance comparison chart of DeepMASLD and its single-modal variant; Figure 6 This is a schematic diagram of the analysis of the top 15 most important features ranked by the F-score in the analysis of variance. Figure 7 This is a schematic diagram of a MASLD data processing system based on multimodal data fusion; Figure 8 This is a schematic diagram of a system deployed based on a B / S architecture. Figure 9 This is a schematic diagram of a multimodal data fusion method. Figure 10 This is a schematic diagram of an electronic device. Detailed Implementation

[0019] The present invention will now be described with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. It should be understood that those skilled in the art can modify the invention described herein while still achieving its advantageous effects. Therefore, the following description should be understood as being of general knowledge to those skilled in the art and is not intended to limit the invention.

[0020] The serial numbers assigned to components in this document, such as "first," "second," etc., are merely used to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages). In the description of this invention, it should be understood that the terms "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention.

[0021] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0022] The invention is described more specifically by way of example in the following paragraphs with reference to the accompanying drawings. The advantages and features of the invention will become clearer from the following description. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the invention.

[0023] Metabolic fatty liver disease (MASLD) is a common chronic liver disease with a rising global prevalence. Early diagnosis and risk assessment of MASLD are crucial for preventing disease progression to liver fibrosis, cirrhosis, and even liver cancer. However, MASLD is often asymptomatic in its early stages, and traditional biochemical tests are often insufficient to detect hepatic steatosis or inflammation.

[0024] Currently, the diagnosis and assessment of MASLD mainly rely on the following methods: 1. Liver biopsy: the gold standard, but it is invasive, costly, and carries the risk of complications; 2. Imaging examinations: such as ultrasound, CT, and MRI, but with limited sensitivity and specificity; 3. Serum biomarkers: such as liver function indicators like ALT and AST, but lack sufficient accuracy; 4. Clinical scoring systems: such as NAFLD fibrosis score, FIB-4 index, etc., but their applicability is limited.

[0025] In recent years, the application of artificial intelligence technology in the medical field has provided new ideas for solving the diagnostic challenges of MASLD. However, most existing technologies are based on single-modal data, making it difficult to fully utilize the complementarity of multi-source information. A few multimodal studies often employ simple feature concatenation methods, failing to fully consider the inherent correlations and differences between different modalities.

[0026] Therefore, a multimodal fusion method specifically designed for MASLD data processing is needed to effectively integrate multiple data sources such as clinical indicators and tongue images, thereby improving the accuracy and reliability of MASLD data processing.

[0027] Based on this, the present invention aims to provide a MASLD data processing scheme based on multimodal data fusion: (See reference...) Figure 1 This invention illustrates a multimodal deep learning framework for MASLD (referred to as the DeepMASLD model in this invention) that combines specific clinical indicators and tongue image features of MASLD. This framework includes: a data input component for acquiring MASLD-related multimodal datasets, which include structured clinical data and tongue image data; a MASLD-specific graph construction component for constructing patient similarity graphs based on clinical features in the clinical data; a dual-branch feature extraction component, including a graph neural network branch and a convolutional neural network branch, where the graph neural network processes the patient similarity graph structure to extract clinical feature representations, and the convolutional neural network processes the tongue image data to extract tongue image feature representations; a multimodal attention fusion component for adaptively fusing clinical features and tongue image features through an attention fusion mechanism to obtain multimodal fused features; and a feature importance analysis and output component for evaluating the contribution of each multimodal fused feature to MASLD data processing, utilizing deep learning technology to achieve efficient MASLD data processing, analysis, and output.

[0028] In practice, the four severity levels of MASLD can be classified as: Normal, Mild, Moderate, and Severe.

[0029] To evaluate the effectiveness of the proposed DeepMASLD multimodal model, this invention compares its performance with several widely used machine learning algorithms, including Support Vector Machine (SVM), Random Forest (RF), Gradient Boosting Machine (GBM), Lightweight Gradient Boosting Machine (LightGBM), and Extreme Gradient Boosting (XGBoost).

[0030] In the evaluation methods, the DeepMASLD model consistently ranked highest in both classification accuracy and F1 score, highlighting its superior ability to perform MASLD risk stratification on complex multimodal data. Ensemble-based models (such as XGBoost and LightGBM) demonstrated competitive performance, while traditional classifiers (such as SVM) showed relatively limited predictive power. These results indicate that using a deep learning framework to integrate the visual features of structured health records and tongue images offers significant advantages over traditional methods.

[0031] refer to Figure 2 As shown, DeepMASLD performs best in both accuracy and F1 score, exceeding 0.90, significantly outperforming traditional machine learning methods. This indicates its ability to capture more complex nonlinear relationships in the feature space, achieving higher classification accuracy. In contrast, SVM performs relatively weakly in prediction tasks, with both accuracy and F1 score at low levels, indicating limited adaptability to high-dimensional features. Ensemble learning methods such as Random Forest, GBM, LGBM, and XGBoost all demonstrate robust performance, with XGBoost achieving both accuracy and F1 score close to 0.88, showing strong generalization ability.

[0032] The results show that the DeepMASLD proposed in this invention achieves performance optimization at the model level, and also provides a comparison and validation basis for multiple algorithms for clinical MASLD risk screening. It can improve the robustness and adaptability of the system while ensuring accuracy, thus having significant clinical promotion and industrialization value.

[0033] Figure 3A The pie chart illustrates the distribution of the MASLD dataset. In terms of data percentage, severe MAFLD accounts for 35.7%, moderate MAFLD for 22.3%, mild MAFLD for 12.9%, and normal for 29.1%. MAFLD stands for Metabolic Associated Fatty Liver Disease.

[0034] Figure 3BThe box plots presented not only demonstrate the numerical distribution characteristics of the controlled attenuation parameter (CAP) across different severity groups, but also visually reveal the scientific rationality and application value of this technology in graded diagnosis. Specifically, the median CAP value gradually increases from the normal group to the severe group, highly consistent with the clinical staging threshold. This provides solid data support for the technical solution of this invention and highlights the matching between parameter design and actual clinical standards. Furthermore, the comparison of classification accuracy shows that the method performs excellently in the normal group (0.94) and the severe group (0.96), indicating strong discriminative ability under extreme conditions. The accuracy in the mild group (0.73) and the moderate group (0.86) is relatively low, mainly due to the strong overlap in their numerical ranges, reflecting the challenge of the algorithm in identifying intermediate states. These results clearly demonstrate the innovation and practicality of the invention in improving the accuracy of early screening and graded diagnosis, laying a solid foundation for related medical imaging diagnostic devices or intelligent auxiliary decision-making systems.

[0035] exist Figures 4A to 4D From the comparison of predicted labels, we can see the distribution of predicted labels (see...). Figure 4B (Illustrative) and the actual label distribution (see) Figure 4A The models are highly similar to each other (as shown in the illustration), resulting in a classification accuracy of up to 92.04%, highlighting the model's strong predictive performance and generalization ability. Furthermore, from the misclassification map (see...) Figure 4C As illustrated (see diagram), most errors occur between adjacent severity categories (e.g., mild vs. moderate, moderate vs. severe), reflecting the inherent difficulty in distinguishing these borderline cases in clinical practice. Furthermore, Figure 4D The model's predictions show a high degree of consistency with the CAP value, further confirming the biological rationale behind the model's output.

[0036] Overall, these results indicate that by integrating tongue image data with structured health examination records, the model can achieve accurate, interpretable, and clinically relevant MASLD risk stratification.

[0037] To evaluate the contributions of different data modalities, this invention also conducted ablation studies and experimentally compared three variants of the DeepMASLD model. These three variants are: DeepMASLD (GNN) using structured clinical data via GNN; DeepMASLD (ResNet-50) using tongue image data via a ResNet-50-based feature extractor; and the complete DeepMASLD model integrating two modalities using an attention-based fusion mechanism. The first two variants are equivalent to unimodal DeepMASLD models, while the last is the complete multimodal DeepMASLD model.

[0038] like Figure 5 As shown, the complete multimodal DeepMASLD model exhibits the best performance, achieving an accuracy of 0.92 and an F1 score of 0.91, demonstrating the advantages of multimodal fusion. In contrast, the unimodal DeepMASLD (GNN) using clinical features achieves an accuracy of 0.90 and an F1 score of 0.87, indicating that structured data itself possesses strong predictive power. However, the unimodal DeepMASLD (ResNet-50) based on tongue images performs significantly worse, with an accuracy of 0.34 and an F1 score of 0.30, indicating that tongue images alone are insufficient for accurate MASLD classification.

[0039] The above results confirm that although structured clinical data provides strong predictive information, integrating visual information of tongue image through ResNet-50 and attention-based fusion mechanism can further improve predictive performance, thus verifying the important value of the DeepMASLD multimodal learning framework of this invention.

[0040] Regarding fusion features, Figure 6 The study presented the top 15 key features ranked by F-scores using analysis of variance (ANOVA). Results showed that metabolic and liver function-related indicators such as triglycerides (TG, 35.90), alanine aminotransferase (ALT, 32.50), and total cholesterol (TC, 20.45) dominated the feature contributions, highlighting their central role in disease diagnosis. Meanwhile, low-density lipoprotein cholesterol (LDLC), mean heart rate, fasting plasma glucose (FPG), and indicators related to kidney and spleen function also demonstrated high importance. This finding further strengthens the scientific basis for multi-system, multi-dimensional collaborative diagnosis.

[0041] In summary, this invention not only clarifies the role mechanism of key biomarkers in the hierarchical diagnosis of MASLD, but also proposes an intelligent screening and prediction architecture based on multi-feature optimization (DeepMASLD model), providing an implementable technical solution for disease risk prediction and precision medicine.

[0042] The invention will now be illustrated by examples.

[0043] Example 1: MASLD Multimodal Data Processing Example This embodiment provides a MASLD data processing method based on multimodal data fusion, specifically for processing MASLD-related clinical indicators and tongue image data.

[0044] The MASLD-related data processed by this invention includes the following two categories: Structured clinical data (Clinical Examination Indicators) include liver function indicators (ALT, AST, GGT, etc.), blood lipid indicators (TG, TC, LDL-C, HDL-C, etc.), glucose metabolism indicators (FPG, HbA1c, etc.), and anthropometric indicators (BMI, waist circumference, etc.). Tongue Image Data: Images of the tongue acquired through standardized acquisition equipment, including visual features such as tongue color, tongue coating thickness, and tongue shape.

[0045] A MASLD data processing method based on multimodal data fusion may include the following steps: Data preprocessing: Standardize clinical data, and normalize, color correct, and segment the background of tongue images; MASLD-specific map construction: A patient similarity map is constructed based on clinical data, and the similarity between patients is calculated using MASLD-related features (such as ALT, TG, BMI, etc.). Clinical Feature Extraction: Graph Attention Network (GAT) is used to extract deep features from clinical data and capture non-linear relationships between features; Tongue image feature extraction: Visual features of tongue images are extracted using convolutional neural networks (CNN), with a focus on tongue image manifestations related to MASLD; Multimodal feature fusion: Clinical features and tongue appearance features are fused through an attention mechanism to obtain a fused multimodal feature representation; Output results: Based on the fusion features, output data processing results such as feature importance analysis, data visualization, or risk assessment reports.

[0046] This embodiment achieves one or more of the following beneficial effects through multimodal data fusion and processing technology for MASLD: fully utilizing MASLD-related clinical indicators and tongue features to improve the targeting of data processing; better capturing the similarity relationship between patients through the construction of MASLD-specific maps; fusion of multimodal information to improve the accuracy and reliability of data processing results; and providing interpretable data processing results, which is helpful for medical research and clinical decision support.

[0047] In this invention, the MASLD specific graph construction algorithm can be illustrated as follows:

[0048] Given N individuals with D-dimensional clinical characteristics, the feature matrix is ​​represented as follows: Construct a patient similarity graph G = (V, E), where each node... For each patient, the weighted cosine similarity of edge E based on MASLD-related features is constructed using the k-nearest neighbor (k-NN) method. The edge weight between nodes i and j is defined as follows: Among them, A ij The edge weights between nodes i and j represent the adjacency relationships between vertices, thus forming the adjacency matrix.

[0049] In this invention, the graph attention network algorithm can be illustrated as follows:

[0050] In modeling graph structures using Graph Attention Networks (GAT), the attention coefficient between node i and its neighbor j is calculated as follows: Where W is the learnable weight matrix, and a is the attention vector. Indicates splicing, It is the activation function. In this example, ELU is used. ELU (Exponential Linear Unit) is a commonly used neural network activation function that can overcome some limitations of the ReLU function.

[0051] In this invention, the feature extraction algorithm can be illustrated by the following implementation example.

[0052] GNN Extraction of Medical Features: This method uses Graph Attention Networks (GAT) to extract deep features from clinical data and capture non-linear relationships between features. In implementation, the model stacks two layers of GAT: the first layer uses eight attention mechanisms, and the second layer uses one. Batch normalization and dropout techniques are applied between the layers.

[0053] It should be noted that the GNN output and loss function are: GNN output node embedding This is used for classification and multimodal fusion. For supervised learning, the cross-entropy loss is minimized. in, It is the predicted probability of category C. It's a real label.

[0054] Visual Feature Extraction Based on CNN: A convolutional neural network (CNN) is used to extract visual features from tongue images, focusing on tongue features related to MASLD (Massively Interpretive Alternatives of Language). In implementation, a pre-trained ResNet-50 backbone network is used to process the tongue images. The original classification head is discarded, and the convolutional feature extractor is retained. The resulting 2048-dimensional features are... Flatten and project onto a 1024-dimensional latent space: in, This is a learnable weight matrix.

[0055] In this invention, the feature fusion algorithm can be illustrated as follows:

[0056] Feature Projection and Alignment: Independently projecting GNN and CNN embeddings into a shared 256-dimensional space using linear transformation and LayerNorm. Among them, ReLU is used as the activation function. Since the gradient of the ReLU function can only take the values ​​of 0 or 1, and ReLU truncates negative values ​​to 0, it introduces sparsity into the network and further improves the computational efficiency.

[0057] Attention-based feature fusion: To adaptively fuse modalities, attention weights are calculated using the following formula. : ; In this process, softmax is used as the activation function to output values ​​for the fused modes.

[0058] The final fusion characterization is given by the following equation: ; Among them, the fusion representation incorporates features from GNN and CNN to better represent the data.

[0059] In this invention, the fused feature output can be achieved using a classifier, as illustrated in the following implementation example.

[0060] Fusion classifier output: Input the fused representation into a multilayer perceptron (MLP) classifier to predict the class label: This MLP consists of three fully connected layers (see reference). Figure 1 (Illustrated Classifier), ReLU activation function and dropout are used between layers.

[0061] The fusion loss is also defined using the classification cross-entropy definition: Among them, minimizing the difference between the model's predicted probability and the actual label by classifying cross-entropy provides a better measure of the difference between the predicted probability and the true label.

[0062] Example 2: MASLD Feature Importance Analysis Example This embodiment provides an example of MASLD feature importance analysis, which identifies the most important features for MASLD data processing based on multimodal data processing.

[0063] In practice, the following steps may be included when performing MASLD feature importance analysis: The F-values ​​of each clinical feature were calculated using analysis of variance (ANOVA) to assess the importance of the feature. The contribution of each modality and feature to the final result is analyzed based on attention weights; By incorporating domain knowledge, the importance of features is explained and verified. Generate feature importance reports to guide subsequent research and applications.

[0064] Based on the above analysis steps, the most important features can be identified in MASLD data processing. These important features include: triglycerides (TG): F value = 35.90; alanine aminotransferase (ALT): F value = 32.50; total cholesterol (TC): F value = 20.45; low-density lipoprotein cholesterol (LDL-C): F value = 18.70; tongue color feature: attention weight = 0.24; tongue coating thickness feature: attention weight = 0.19. The feature importance analysis data can be found by referring to... Figure 6 This is just an illustration; no further explanation will be provided.

[0065] In some examples, the top N most important features are selected based on the F-value of the analysis of variance, where N can preferably be 15-20.

[0066] In some examples, key features may include at least three of the following: triglycerides (TM), alanine aminotransferase (ALT), total cholesterol (TC), low-density lipoprotein (LDL-C), average heart rate, and fasting blood glucose.

[0067] This embodiment achieves one or more of the following beneficial effects through feature importance analysis: identifying the most relevant features in MASLD data processing and improving processing efficiency; providing clues for MASLD mechanism research and promoting medical knowledge discovery; guiding clinical data collection and optimizing resource allocation; and enhancing the interpretability of data processing results and increasing user trust.

[0068] Example 3: Implementation Example of MASLD Processing System Based on Multimodal Data Fusion refer to Figure 7 This illustrates a MASLD processing system based on multimodal data fusion, which may include the following data modules: Data acquisition module 101 is configured to acquire MASLD-related multimodal datasets, the datasets including structured clinical data and tongue image data; The graph construction module 103 is configured to construct a patient similarity graph structure based on the MASLD-specific clinical features in the clinical data. The feature extraction module 105 is configured to use a graph neural network to extract clinical features from the graph structure and to use a convolutional neural network to extract tongue features from the tongue image data. The fusion module 107 is configured to fuse the clinical feature representation and the tongue image feature representation through an attention fusion mechanism to obtain multimodal fusion features; Output module 109 is configured to output MASLD data processing results based on the multimodal fusion features.

[0069] The system workflow in this embodiment is illustrated as follows: Based on the data acquisition module's function, the user uploads a tongue image and fills out a health questionnaire. The system can also preprocess and standardize the uploaded data. The system calls a pre-trained DeepMASLD model for prediction. The DeepMASLD model can include relevant modules with data processing capabilities, such as a graph construction module, a feature extraction module, and a fusion module, thereby utilizing these modules to complete data processing. The system returns risk level assessment results and recommendations through the output module.

[0070] The system provided in this embodiment has one or more of the following advantages: user-friendly interface and easy operation; fast system response and real-time prediction capability; support for concurrent access by multiple users, making it suitable for use in medical institutions; and provides detailed risk assessment reports to assist doctors in diagnosis.

[0071] It should be noted that the aforementioned multiple data modules may be located in the same device or in different devices.

[0072] Below is another implementation example based on a B / S architecture deployment.

[0073] refer to Figure 8This illustration illustrates the deployment of local data upload devices (such as computers) in primary healthcare institutions. Users can then upload tongue images and health questionnaire data via a web interface on their computers. The backend server runs a DeepMASLD model to process the data and returns a risk level recommendation based on the fused processing results. In this embodiment, the system adopts a B / S architecture (Browser / Server). The front end provides data upload through a browser, which, due to its lack of performance requirements, can be flexibly deployed in primary healthcare institutions with limited medical facilities. The backend uses a server architecture to deploy data processing and prediction models for centralized data processing. Finally, the results returned by the server are displayed in the user's browser. For example, the multimodal data processing results and key data such as feature importance can be visualized through the browser to assist doctors in diagnosis.

[0074] By adopting a B / S architecture, not only can non-invasive screening be achieved, making it suitable for large-scale population screening, but system deployment costs can also be reduced, making it ideal for promotion and application in areas with underdeveloped medical facilities. Furthermore, deployment based on a multimodal fusion solution can improve accuracy to over 92%, effectively conducting data collection in areas with underdeveloped medical facilities and improving data comprehensiveness and accuracy.

[0075] Example 4: A general multimodal data fusion scheme Based on the above examples, this embodiment also provides a multimodal data fusion scheme with wider applicability.

[0076] refer to Figure 9 In conjunction with the aforementioned Figure 1 This embodiment illustrates a multimodal data fusion method, which may include the following steps: S202. Obtain a multimodal dataset, which includes structured data of the first modality and image data of the second modality; wherein the structured data of the first modality and the image data of the second modality are different types of data with complementary information, such as medical clinical data and tongue image data in the aforementioned example, so as to serve as multimodal data, which facilitates the full utilization of the complementary information between the multimodal data through subsequent fusion.

[0077] S204. Construct a data relationship graph structure based on the structured data; wherein, the data relationship graph is a graph that facilitates feature processing of the structured data, such as a proximity graph G(V, E) representing similarity, where V represents a node and E represents the edge weight between nodes. For details, please refer to the aforementioned example, which will not be elaborated further.

[0078] S206. Use a graph neural network to process the data relationship graph structure and extract a first feature representation; wherein the first feature representation is used to characterize the unique attributes of each data node in the data relationship graph structure corresponding to the structured data of the first modality. For example, in the aforementioned example, the first feature representation adopts the similar medical characteristics of patients in clinical practice.

[0079] S208. Process the image data using a convolutional neural network to extract a second feature representation; wherein the second feature representation is used to characterize the unique attributes of each data in the image data of the second modality, such as in the aforementioned example, the second feature representation uses visual features similar to those in the patient's tongue image data.

[0080] S210. The first feature representation and the second feature representation are fused through an attention fusion mechanism to obtain a multimodal fusion feature; S212. Based on the multimodal fusion features, output the data processing results.

[0081] It should be noted that this embodiment is a general fusion solution based on the aforementioned MASLD example. Therefore, the corresponding MASLD example content can be referred to in the implementation, and will not be elaborated here.

[0082] For example, constructing a data relationship graph structure based on structured data includes: Calculate the similarity between different data samples; The connection relationships between data samples are determined based on similarity thresholds or the k-nearest neighbor algorithm; Construct a graph structure G=(V, E), where V represents the set of nodes, each node corresponds to a data sample, and E represents the set of edges, with edge weights calculated based on similarity. For example, attention fusion mechanisms include: Project the first feature representation and the second feature representation onto the same feature space; Calculate the attention weights for each modality feature; Multimodal fusion features are obtained by weighting and fusing modal features based on attention weights.

[0083] Example 5: Staged Training Example of Multimodal Data Fusion The training process implemented at the DeepMASLD model level in the aforementioned example can adopt a two-stage training strategy: the first stage trains the graph neural network module separately; the second stage fixes the graph neural network parameters and trains the feature fusion module and the output module.

[0084] Specifically, a multimodal training dataset, including structured data and image data, is first obtained. Then, a two-stage training strategy is implemented to train the multimodal fusion model: in the first stage, only the structured data is used to train the graph neural network module; in the second stage, the graph neural network parameters are fixed, and the feature fusion module and output module are trained using the multimodal data.

[0085] It should be noted that the first stage of training the graph neural network module includes: constructing a data relationship graph based on structured data, and using the graph neural network to process the data relationship graph and extract feature representations to facilitate prediction task training based on the extracted features. Furthermore, in the second stage of training feature fusion and output, a pre-trained graph neural network is used to extract structured feature representations, and a convolutional neural network is used to extract image feature representations. Finally, multimodal features are fused through an attention fusion mechanism, and prediction task training is performed based on the fused features.

[0086] In some implementations, the two-stage training strategy uses different learning rates, for example, the first stage learning rate is 0.001-0.005 and the second stage learning rate is 0.0001-0.0005.

[0087] In some implementations, the training method also includes an early stopping strategy, which stops training when the performance of the validation set does not improve over multiple consecutive training rounds.

[0088] Example 6: Implementation Examples of Electronic Devices and Storage Media This embodiment provides an electronic device, including: At least one processor; Memory, which stores computer programs; When the computer program is executed by the processor, it implements the method as described in any one of the present invention, such as the general fusion method in the foregoing examples, such as the MASLD data processing method in the foregoing examples, etc.

[0089] like Figure 10 As shown in the figure, the present invention also provides a structural schematic diagram of an electronic device, which illustrates the structure of the electronic device 500. Here, the electronic device 500 is merely an example and should not be construed as limiting the function and scope of use of the embodiments of the present invention.

[0090] The electronic device 500 may include: at least one processor 510; and a memory 520 communicatively connected to the at least one processor; wherein the memory 520 stores instructions executable by the at least one processor 510, the instructions being executed by the at least one processor 510 to enable the at least one processor 510 to perform: the container damage quantification method based on geometric calibration as described in any embodiment of this application.

[0091] It should be noted that the electronic device 500 can be represented in the form of a general-purpose computing device, such as a server device.

[0092] In practice, the components of electronic device 500 may include, but are not limited to: at least one processor 510, at least one memory 520, and a bus 530 connecting different system components (including memory 520 and processor 510), wherein bus 530 may include a data bus, an address bus, and a control bus.

[0093] In practice, memory 520 may include volatile memory, such as random access memory (RAM) 5201 and / or cache memory 5202, and may further include read-only memory (ROM) 5203.

[0094] The memory 520 may also include a program tool 5205 having a set (at least one) of program modules 5204, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0095] The processor 510 performs various functional applications and data processing by running computer programs stored in the memory 520.

[0096] Electronic device 500 can also communicate with one or more external devices 540 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 560, which communicates with other modules in electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0097] This embodiment provides a computer storage medium storing computer-executable instructions, which are configured as described in any embodiment of the present invention.

[0098] It should be noted that the computer storage medium may include, but is not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0099] In possible implementations, the present invention may also provide the data processing as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform several steps of the method described in any of the foregoing embodiments.

[0100] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0101] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of equivalent technology, this invention is also intended to include these modifications and variations.

Claims

1. A multi-modal data fusion method, characterized in that, The method comprises the following steps: obtaining a multi-modal data set comprising structured data of a first modality and image data of a second modality; constructing a data relationship graph structure based on the structured data; processing the data relationship graph structure using a graph neural network to extract a first feature representation; processing the image data using a convolutional neural network to extract a second feature representation; fusing the first feature representation and the second feature representation through an attention fusion mechanism to obtain a multi-modal fusion feature; outputting a data processing result based on the multi-modal fusion feature.

2. The method of claim 1, wherein, The constructing of the data relationship graph structure based on the structured data comprises: calculating the similarity between different data samples; determining the connection relationship between data samples based on a similarity threshold or a k-nearest neighbor algorithm; constructing a graph structure G=(V, E), wherein V represents a node set, each node corresponds to a data sample, E represents an edge set, and the edge weight is based on the similarity calculation; and / or, the attention fusion mechanism comprises: projecting the first feature representation and the second feature representation into the same feature space; preferably, embedding the graph neural network and the convolutional neural network into the same shared feature space through linear transformation and LayerNorm; calculating attention weights of each modality feature; preferably, by the following formula calculating attention weights to adaptively fuse modalities, wherein , the activation function is softmax, W1, W2 are attention weights, 、 is a feature projection; weighting and fusing the modal features according to the attention weight to obtain a multi-modal fusion feature; preferably, adopting a multi-layer perceptron classifier to fuse and predict the class label, and further adopting a multi-layer perceptron classifier preferably composed of three fully connected layers as a classifier; and / or, after fusion, the method further comprises: combining domain knowledge to explain and verify the feature importance, and generating a feature importance output result.

3. The method of claim 1, wherein, The method further comprises a training process, adopting a two-stage training strategy: the first stage separately trains the graph neural network module; preferably, in the first stage, a data relationship graph is constructed based on the structured data, and a feature representation is extracted by using the graph neural network to process the data relationship graph, and a prediction task is trained based on the extracted feature; the second stage fixes the graph neural network parameters, and trains the feature fusion module and the output module; preferably, in the second stage, a structured feature representation is extracted using a pre-trained graph neural network, and an image feature representation is extracted using a convolutional neural network, and finally a multi-modal feature is fused through an attention fusion mechanism, and a prediction task is trained based on the fused multi-modal feature.

4. A multi-modal data fusion system, characterized by, comprising: a data acquisition module configured to obtain a multi-modal data set comprising structured data of a first modality and image data of a second modality; a graph construction module configured to construct a data relationship graph based on the structured data; a graph neural network module configured to process the data relationship graph structure using a graph neural network to extract a first feature representation; a convolutional neural network module configured to process the image data using a convolutional neural network to extract a second feature representation; a fusion module configured to fuse the first feature representation and the second feature representation through an attention fusion mechanism to obtain a multi-modal fusion feature; an output module configured to output a data processing result based on the multi-modal fusion feature.

5. A MASLD data processing method based on multi-modal data fusion, characterized in that, comprising the following steps: obtaining a MASLD-related multi-modal dataset, the dataset comprising structured clinical data and tongue image data; constructing a patient similarity graph structure based on MASLD-specific clinical features in the clinical data; processing the graph structure using a graph neural network to extract clinical feature representations; processing the tongue image data using a convolutional neural network to extract tongue feature representations; fusing the clinical feature representations and tongue feature representations through an attention fusion mechanism to obtain multi-modal fusion features; outputting a MASLD data processing result based on the multi-modal fusion features.

6. The method of claim 5, wherein, The constructing of the patient similarity graph structure based on the MASLD-specific clinical features comprises: selecting a subset of clinical features related to MASLD; assigning weights to different features based on feature importance; calculating the similarity between patients using a weighted similarity calculation method; constructing a graph structure based on a similarity threshold or a k-nearest neighbor algorithm; and / or, the attention fusion mechanism comprises: projecting the clinical feature representations and visual feature representations into the same feature space; calculating the attention weights of the modal features, wherein the initial weight of the clinical features is 0.5-0.7, and the initial weight of the visual features is 0.3-0.5; weighting and fusing the modal features according to the attention weights to obtain multi-modal fusion features; and / or, the graph neural network is a graph attention network, which includes a two-layer structure, the first layer uses 6-10 attention heads, and the second layer uses 1 attention head, and batch normalization and dropout techniques are used between layers; and / or, the convolutional neural network is a pre-trained ResNet-50 model, which is fine-tuned for tongue image features to extract tongue color, tongue fur thickness and tongue shape features; and / or, the structured clinical data includes at least one of liver function indicators, blood lipid indicators, blood glucose metabolism indicators and anthropometric indicators.

7. A multi-modal data fusion based MASLD processing system, characterized in that, comprises: a data acquisition module configured to obtain a MASLD-related multi-modal dataset, the dataset comprising structured clinical data and tongue image data; a graph construction module configured to construct a patient similarity graph structure based on MASLD-specific clinical features in the clinical data; a feature extraction module configured to extract clinical features in the graph structure using a graph neural network and tongue features in the tongue image data using a convolutional neural network; a fusion module configured to fuse the clinical feature representations and tongue feature representations through an attention fusion mechanism to obtain multi-modal fusion features; an output module configured to output a MASLD data processing result based on the multi-modal fusion features.

8. A multi-modal data processing system based on B / S architecture, characterized in that, comprises: a browser end for uploading a dataset required by the system locally, the dataset comprising first modal data and second modal data; A server is configured to run a preset multi-modal fusion model to complete data processing and return the fusion processing result to the browser; wherein the multi-modal fusion model comprises: a graph construction component configured to construct a similarity structure graph based on first modal data; a double-branch feature extraction component comprising a graph neural network branch and a convolutional neural network branch, the graph neural network being configured to process the similarity graph structure and extract a first feature representation, and the convolutional neural network being configured to process second modal data and extract a second feature representation; a multi-modal attention fusion component configured to adaptively fuse the first feature representation and the second feature representation through an attention fusion mechanism to obtain multi-modal fusion features; and a feature importance analysis and output component configured to evaluate the contribution of each multi-modal fusion feature.

9. An electronic device, comprising: Comprise: at least one processor; a memory storing a computer program; when the computer program is executed by the processor, the method of any one of claims 1-3 is implemented, or the method of any one of claims 5-6 is implemented.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-3, or the method of any one of claims 5-6.

Images