IgAN prediction method, system and device based on multi-modal fusion

Abstract

The invention discloses an IgAN prediction method, system and device based on multi-modal fusion, and belongs to the technical field of image data processing, and the method comprises the steps: collecting to-be-predicted data; predicting the to-be-predicted data through a prediction model based on multi-modal fusion to obtain a fifth prediction result; wherein the network of the prediction model comprises the following branches: a tongue picture branch, a face branch, a table branch, a fusion branch, a gating module and a fifth prediction module. Different branches are integrated through an expert hybrid integration mechanism, multi-modal fusion of to-be-predicted data is achieved, under the condition that part of data is missing, a gating module can select available experts for prediction, and stable prediction performance is kept; the method effectively integrates the clinical data, the face image, the tongue image and other heterogeneous modal data while maintaining the independent contribution of each modal data, and achieves better prediction performance through complementary information extraction and hierarchical fusion strategies.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and more specifically to an IgAN prediction method, system, and apparatus based on multimodal fusion. Background Technology

[0002] Immunoglobulin A nephropathy (IgAN) is one of the most common causes of primary glomerulonephritis worldwide. Early diagnosis of IgAN is crucial for timely and effective intervention and for slowing disease progression.

[0003] IgAN presents with highly heterogeneous clinical manifestations; 40–50% of cases present only with asymptomatic microscopic hematuria and stable renal function, and currently no rigorously validated serum or urinary biomarkers have been identified, making clinical identification difficult. The gold standard for diagnosing IgAN is renal biopsy, but this is invasive and has many limitations. In situations where renal biopsy is not feasible, in real clinical practice, the diagnosis of IgAN primarily relies on a process of elimination, requiring a systematic differentiation and exclusion of other kidney diseases with similar clinical presentations.

[0004] Integrating clinical data using machine learning to train IgAN prediction models and then using these models to predict IgAN disease has become an important research direction. Patent CN116705293A discloses a prediction model and method for IgA nephropathy combining traditional Chinese and Western medicine. Through a specially designed artificial neural network model incorporating both Chinese and Western medicine, it can quickly perform modeling and extrapolation calculations to predict the risk of IgA nephropathy progression. However, traditional Chinese medicine has long relied on facial and tongue features as diagnostic criteria, features that this patent cannot address.

[0005] Therefore, there is a need to design an IgAN prediction method, system, and device based on clinical data and multimodal fusion of tongue and facial features. Summary of the Invention

[0006] To address the aforementioned technical problems in the existing technology, this invention provides an IgAN prediction method, system, and device based on multimodal fusion, which predicts IgAN based on the multimodal fusion of clinical data, facial features, and tongue features.

[0007] The first aspect of this invention discloses an IgAN prediction method based on multimodal fusion, comprising the following steps: Collect the data to be predicted; use a prediction model based on multimodal fusion to predict the data to be predicted, and obtain the fifth prediction result; The prediction model's network includes the following branches: tongue image branch, face branch, table branch, fusion branch, gating module, and fifth prediction module. The tongue image branch is used to extract tongue image features from the tongue image image; and a first expert predicts the tongue image image to obtain a first prediction result; The facial branch is used to extract facial features from a facial image and to predict the facial image by a second expert to obtain a second prediction result; The table branch is used to extract table features from clinical data based on graph neural networks, and to obtain a third prediction result by using a third expert to predict the clinical data. The fusion branch is used to fuse tongue features, facial features, and table features to obtain a third fusion feature, and a fourth expert predicts the third fusion feature to obtain a fourth prediction result. The gating module is used to select important experts from the first expert, the second expert, the third expert, and the fourth expert; The fifth prediction module is used to obtain a fifth prediction result based on the prediction results of the key experts.

[0008] Preferably, the facial branch includes a second feature extraction module, a clustering graph attention module, and a second prediction module; The facial image is input into the second feature extraction module, which is connected to the aggregation graph attention module; the output values ​​of the second feature extraction module and the aggregation graph attention module are concatenated and then pooled to obtain the second concatenated feature. The second concatenated feature is then subjected to global average pooling to obtain the second classification feature; On the one hand, the second classification feature is predicted by the second prediction module; on the other hand, facial features are obtained after the second classification feature is projected.

[0009] Preferably, the tongue branch includes a first feature extraction module, a Transformer module, and a first prediction module; The tongue image is processed by the first feature extraction module and the Transformer module to obtain the first classification features; On the one hand, the first classification feature is predicted by the first prediction module to obtain the first prediction result; On the other hand, after the first classification features pass through the first projection layer, tongue image features are obtained. ； The first feature extraction module includes a fifth convolutional layer, a first spatial attention submodule, a sixth convolutional layer, a second spatial attention submodule, a seventh convolutional layer, a third spatial attention submodule, an eighth convolutional layer, and a pooling operation connected in sequence. The spatial attention submodule includes: concatenating the first feature map Fl after average pooling and max pooling respectively, passing it through a convolutional layer and activation function, and concatenating the resulting feature map with the first feature map by dot product to obtain the second feature map.

[0010] Preferably, the table branches include a graph neural network submodule, a gradient boosting submodule based on the XGBoost method, and a self-attention submodule; The graph neural network submodule is sequentially connected to the self-attention submodule and the linear layer to obtain the fourth feature; The XGB features extracted by the gradient boosting submodule are concatenated with the fourth feature after passing through the projection layer, and then passed through the fourth projection layer to obtain the table features. The table features are predicted by the third prediction module to obtain the third prediction result.

[0011] Preferably, the table branches include a graph neural network submodule, a gradient boosting submodule based on the XGBoost method, and a self-attention submodule; The tabular data is connected to the self-attention submodule; On the one hand, the output of the self-attention submodule is connected to the graph neural network submodule to obtain the fourth feature; On the other hand, after the output of the attention submodule is connected to the gradient boosting submodule and the third projection layer, it is concatenated with the fourth feature, and then passed through the linear layer and the fourth projection layer to obtain the table feature; The table features are predicted by the third prediction module to obtain the third prediction result; Among them, table features h tab Represented as: ; in, NTP() For linear layer calculation, GCN() Graph neural network computation; MLP() Calculations for the projection layer, L Characterized by XGB SELU() Calculations are performed for the fourth projection layer; This is the output value of the self-attention submodule.

[0012] Preferably, the fusion branch includes a first fusion submodule and a second fusion submodule. The first fusion submodule is used to fuse two sub-features to obtain a fused sub-feature, wherein the sub-feature is selected from tongue image features, table features and facial features; The second fusion submodule is used to fuse the first fusion feature and the second fusion feature to obtain the third fusion feature; The fourth prediction module is used to predict the third fusion feature to obtain a fourth prediction result.

[0013] Preferably, the first fusion submodule projects the sub-features onto the projection space; the projection space includes a coarse-level projection layer, a medium-level projection layer, and a fine-level projection layer; The fourth concatenated feature is obtained by multiplying the projected features of the corresponding projection layers of the two sub-features. The fourth concatenation feature is scale-specific upsampling, followed by power set concatenation operation, and then passed through a linear layer to obtain the fifth concatenation feature; The precise projection features of the two sub-features are stitched together, and then fused with the fifth stitched feature after passing through the Transformer module to obtain the fused sub-feature; The two sub-features include tongue image features and table features, and the fusion sub-features include the first fusion feature; Alternatively, the two sub-features include facial features and table features, and the fused sub-features include a second fused feature.

[0014] Preferably, the probability of the fifth prediction result is expressed as follows: ; in, x This is represented as the input sample. h concat This is a concatenated vector obtained by encoding the various modalities of the input sample separately. W gate Represented as gating weight, g The importance score vector for experts; top-2(g) This indicates that the two experts with the highest scores will be selected; g top-2 This represents the importance score vector of the two experts with the highest scores. softmax() For normalization operations, p k This represents the prediction probability of the k-th expert; This represents the probability of the fifth prediction result.

[0015] A second aspect of the present invention provides a prediction system for implementing the above-described IgAN prediction method, comprising an input module and a sixth prediction module. The input module is used to obtain the data to be predicted; The sixth prediction module is used to predict the data to be predicted using a prediction model based on multimodal fusion, and obtain the fifth prediction result.

[0016] A third aspect of the present invention provides a prediction apparatus, including a memory storing code, which, when executed, implements the IgAN prediction method described above.

[0017] Compared with existing technologies, the beneficial effects of this invention are as follows: by integrating different branches through an expert hybrid integration mechanism, multimodal fusion of the data to be predicted is achieved. In the case of missing data, the gating module can select available experts for prediction, maintaining robust prediction performance. While maintaining the independent contribution of each modality of data, it effectively integrates heterogeneous modality data such as clinical data, facial images, and tongue images. Through complementary information extraction and hierarchical fusion strategies, it achieves better prediction performance. Attached Figure Description

[0018] Figure 1 This is a flowchart of the IgAN prediction method based on multimodal fusion of the present invention; Figure 2 It is a logical block diagram of the prediction model and its network; Figure 3 It is a network diagram of facial branches; Figure 4 This is the network diagram of the second feature extraction module; Figure 5 This is a network diagram of the branches of the tongue; Figure 6 This is the network diagram of the first feature extraction module; Figure 7 It is a type of network diagram with branching tables; Figure 8 It is another type of network diagram with branching tables; Figure 9 It is a network diagram of fused branches; Figure 10 This is the network diagram of the first fusion submodule; Figure 11 This is the logic block diagram of the gating module and the fifth prediction module; Figure 12 This is the ROC curve of the test set; Figure 13 It is the ROC curve of the follow-up set; Figure 14 This is the ROC curve of the prospect set. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The present invention will now be described in further detail with reference to the accompanying drawings: Example 1 provides an IgAN prediction method based on multimodal fusion, such as... Figure 1 This includes the following steps: Step 101: Collect the data to be predicted, which includes clinical data, facial images, and tongue images.

[0021] Step 102: The prediction data to be predicted is predicted using the IgAN-Seeker prediction model based on multimodal fusion to obtain the fifth prediction result. The fifth prediction result includes IgAN classification and prediction probability. The IgAN classification includes whether there is IgAN or no IgAN.

[0022] Among them, such as Figure 2 The prediction model and its network include the following branches: tongue image branch 11, face branch 12, table branch 13, fusion branch 14, gating module 15, and fifth prediction module 16. The tongue image branch 11 is used to extract tongue image features from the tongue image image; and a first expert makes a prediction on the tongue image image to obtain a first prediction result.

[0023] The facial branch 12 is used to extract facial features from the facial image and to predict the facial image by a second expert to obtain a second prediction result; The table branch 13 is used to extract table features from clinical data based on graph neural networks, and to obtain a third prediction result by using a third expert to predict the clinical data.

[0024] The fusion branch 14 is used to fuse tongue features, facial features and table features to obtain a third fusion feature, and a fourth expert predicts the third fusion feature to obtain a fourth prediction result.

[0025] The gating module 15 is used to select an important expert from the first expert, the second expert, the third expert, and the fourth expert; The fifth prediction module 16 is used to obtain a fifth prediction result based on the prediction results of important experts.

[0026] This invention integrates different branches through an expert hybrid integration mechanism to achieve multimodal fusion of the data to be predicted. In the case of missing data, the gating module can select available experts for prediction to maintain robust prediction performance. While maintaining the independent contribution of each modality of data, it effectively integrates heterogeneous modality data such as clinical data, facial images, and tongue images. Through complementary information extraction and hierarchical fusion strategies, it achieves better prediction accuracy.

[0027] Example 2 provides a prediction network based on multimodal fusion.

[0028] The facial branching network includes a second feature extraction module, a clustering map attention module, and a second prediction module, such as... Figure 3 .

[0029] The facial image is input to the second feature extraction module, and the output of the second feature extraction module is connected to the aggregation map attention module. The output values ​​of the second feature extraction module and the aggregation map attention module are concatenated and then pooled to obtain the second concatenated feature.F attended Second splicing feature F attended After global average pooling, the second classification feature is obtained. Z face On the one hand, secondary classification features Z face Prediction is performed through the second prediction module, and on the other hand, the second classification features... Z face After projection, facial features are obtained. h face .

[0030] The second expert includes a second prediction module, which comprises a second classifier and second prediction logic. The second classifier can be based on methods such as random forest, Adaboost, and SVM, which are existing technologies and will not be elaborated upon in this invention. The second classifier can predict a second probability, and the second prediction logic includes a second threshold. Based on the second probability and the second threshold, a second prediction result, i.e., the IgAN category and probability, can be obtained.

[0031] Among them, such as Figure 4 The second feature extraction module extracts hierarchical features based on the ResNet-18 backbone network, including four sequentially connected convolutional layers conv1, conv2, conv3, and conv4, and outputs the fourth feature. F conv4 This advanced feature can capture facial features.

[0032] The focus image attention module generates an interpretable attention mask M through progressive optimization. focus This includes performing the following operations in sequence: Conv2D, batch normalization, ReLU activation, Conv2D, batch normalization, ReLU activation, Conv2D, resizing, and softmax activation.

[0033] Attention masking M focus It can be represented as: (1); in, It contains three convolutional layers, and the specific operations and parameters are as follows: .

[0034] Attention masking M focus Bilinear upsampling upsample After aligning to the feature dimension, then with the fourth feature. F conv4 Perform matrix dot product to obtain the second concatenation feature. Fattended , represented as: (2).

[0035] The second classification feature is obtained after global average pooling. Z face , Then the second classification feature Z face Facial features are obtained through the second projection layer of the face. h face , .

[0036] Secondary classification characteristics Z face The second prediction result is obtained by using the second prediction module.

[0037] like Figure 5 The tongue branch network (TNet, spatial-transform encoder) includes: a first feature extraction module, a Transformer module, and a first prediction module.

[0038] The tongue image is processed by the first feature extraction module and the Transformer module to obtain the first classification features. h trans On the one hand, the first classification feature h trans The first prediction module performs prediction to obtain the first prediction result; on the other hand, the first classification features... h trans After passing through the first projection layer, tongue image features are obtained. h tongue .

[0039] like Figure 6 The first feature extraction module includes a fifth convolutional layer, a first spatial attention submodule, a sixth convolutional layer, a second spatial attention submodule, a seventh convolutional layer, a third spatial attention submodule, an eighth convolutional layer, and a pooling operation connected in sequence to obtain the third feature. Z tongue .

[0040] The operations of the spatial attention submodule include: processing the first feature map F l Average pooling AvegPool and max pooling MaxPool The features are then concatenated, passed through convolutional layers and activation functions, and obtained. M spatial The second feature map is obtained by concatenating it with the first feature map through a dot product. .

[0041] Feature map extracted from the eighth convolutional layer Ftongue The third feature is obtained through global average pooling. Z tongue , .

[0042] Feature maps extracted from the first convolutional layer F l For example, The operation formula for the first spatial attention submodule is as follows: ; f 7 7 This indicates a convolutional layer with a kernel size of 7. AvegPool() This is represented as average pooling. MaxPool() For max pooling, This is the activation function. Represented as the second feature map; F l This is the first feature map. This is represented as a dot product.

[0043] The Transformer module uses a two-layer Transformer encoder with eight attention heads to capture long-range dependencies: h trans = TransformerEncoder( z tongue + P pos (5); P pos Represented as a learnable positional embedding; h trans This is represented as the first classification feature, and TransformerEncoder() represents the Transformer encoder. z tongue This is the third characteristic.

[0044] The first classification feature is obtained through prediction by the first prediction module / first expert, resulting in a first prediction result; on the other hand, tongue image features are obtained through projection. h tongue .

[0045] like Figure 7In one specific embodiment, the table branch includes a graph neural network submodule, a gradient boosting submodule based on the XGBoost method, and a self-attention submodule. The graph neural network submodule is sequentially connected to the self-attention submodule and a linear layer to obtain a fourth feature. The XGBoost feature extracted by the gradient boosting submodule is concatenated with the fourth feature after passing through a projection layer, and then passed through the fourth projection layer to obtain the table feature. h tab Table features h tab The third prediction result is obtained through the third prediction module.

[0046] like Figure 8 In another specific embodiment, the tabular data is connected to the self-attention submodule. On one hand, the output of the self-attention submodule is connected to the graph neural network submodule to obtain the fourth feature. On the other hand, the output of the self-attention submodule is connected to the gradient boosting submodule and the third projection layer, then concatenated with the fourth feature, and finally passed through a linear layer and the fourth projection layer to obtain the tabular feature. h tab .

[0047] Specifically, the self-attention submodule includes a multi-head attention mechanism and residual connections and normalization. The formula for the self-attention submodule is expressed as: (6) ； X This represents the input values ​​for the self-attention submodule, specifically tabular data. The output value of the self-attention submodule. This represents a trainable importance weight vector. For bias terms, These are the model parameters.

[0048] Extracting XGB features via a gradient boosting submodule: hyperparameters L XGBoost extracts decision patterns through leaf node indexes: (7); in, L The value of the loss function is a measure of the difference between the predicted and actual values. l This represents the loss function value for each sample, and N represents the number of samples.

[0049] Graph Neural Network (GCN) processing models capture feature relationships through message passing mechanisms; the GCN layer executes this: (8); in, It is an adjacency matrix containing self-loops. It is a degree matrix. It is the square root of the reciprocal of the diagonal elements of the degree matrix. H (l) For the first l The node feature matrix of the layer, It is layer weight. This is the activation function.

[0050] Table features h tab Represented as: (9); in, NTP() For linear layer calculation, specifically ,pass The scaling factor ensures the stability of the gradient flow. GCN() Graph neural network computation; MLP() Calculations for the projection layer, L Characterized by XGB SELU() Calculated for the fourth projection layer.

[0051] like Figure 9 The fusion branch includes a first fusion submodule and a second fusion submodule. The first fusion submodule is used to fuse any two sub-features to obtain a fused sub-feature, wherein the sub-features are selected from tongue image features, table features, and facial features. Specifically, tongue image features and table features are fused to obtain the first fused feature; and this is used to fuse facial features and table features to obtain the second fused feature; however, it is not limited to this, and tongue image features and facial features can also be fused.

[0052] The second fusion submodule is used to fuse the first fusion feature and the second fusion feature to obtain the third fusion feature. The fourth prediction module is used to predict the third fusion feature to obtain the fourth prediction result.

[0053] Specifically, such as Figure 10 The first fusion submodule projects the sub-features onto a projection space s, the projection space s including a coarse-level projection layer. coarse Intermediate projection layer medium and precision projection layer fine The first and second sub-features are projected into the projection space respectively to obtain the projected feature set. Ts ,include T coarse , T medium and T fine The corresponding dimension is .

[0054] Ts = W s1 h s (10); Where Ts represents the projected feature set, W s1 Represented as a weight matrix; h s This is represented as a sub-feature.

[0055] Multiply the projected features of the corresponding projection layers of the two sub-features to obtain the fourth concatenated feature Fs: Fs = TS1 TS2 (11); TS1 represents the projected feature of one sub-feature, and TS2 represents the projected feature of another sub-feature.

[0056] The fourth concatenated feature is scale-specifically upsampled, and a power set concatenation operation is performed. The fifth concatenated feature is then obtained through a linear layer.

[0057] The precision-level projection features of the two sub-features are stitched together, and then fused with the fifth stitching feature through the Transformer module to obtain the fused sub-feature, namely the first fused feature or the second fused feature.

[0058] Specifically, the first fusion feature is the tongue-table fusion feature, and the second fusion feature is the face-table fusion feature. The formula for the first fusion feature is as follows: (12); For the precision-level projection features of the table features, For the precise projection features of the tongue image, [;] represents the concatenation operation of the two features, and MHA() represents the Transformer module operation; F s The fourth concatenated feature is the dot product of the projected features of the table features and the tongue image features. Ups() This is represented as an upsampling operation. This refers to power set splicing, specifically a splicing operation within the power set range of the projection space s. MLP1() This is represented as a linear layer operation.

[0059] The fusion mechanism of the second fusion feature is the same as that of the first fusion feature, and will not be described again in this invention.

[0060] First fusion feature h f1 Second fusion features h f2 splicing operation Then, calculations are performed through the projection layer. MLP final () Obtain the third fusion feature H fusion The fusion method is represented as: H fusion = MLP final (hf1 hf2) (13); in, This represents a splicing operation. hf1 This is the first fusion feature; hf2 This is the second fusion feature; MLP final () Calculations for the projection layer, H fusion This is the third fusion feature.

[0061] Four experts are selected through the four branches mentioned above. The first expert is skilled at analyzing tongue photographs, including texture features such as tongue shape and color. The second expert is skilled at analyzing facial photographs, including appearance features such as complexion and swelling. The third expert is skilled at analyzing clinical data such as urinary red blood cell count, phospholipase A2 receptor antibody, and serum albumin. The fourth expert is skilled at comprehensively analyzing the above data.

[0062] like Figure 11 The gating module 15 is used to dynamically select the two most relevant expert / prediction results for each sample, and the fifth prediction module 16 is used to calculate the final prediction probability, i.e., the fifth prediction result. (14); in, W gate The gating weight is represented by x, which represents the input sample and can contain multiple modalities of data: clinical data, tongue images, and facial images. h concat A concatenated vector consisting of encoded modal data of the input sample; g Let g be a four-dimensional vector representing the importance score of each expert. For example, g[0.8,2.1,1.5,3.0] indicates that the fourth expert (fusion expert) is the most important, followed by the second expert, and so on. top-2(g) This method selects the two experts with the highest scores; it can reduce noise interference and avoid weak experts from creating noise; it improves prediction efficiency by activating and using only some experts and their related models, and can achieve sparsity and adaptability. g top-2 This indicates the importance scores of the two experts with the highest scores; softmax() For the normalization operation, k represents the kth expert. p k This represents the prediction probability of the k-th expert; This is represented as the final predicted probability.

[0063] The fifth prediction module obtains the IgAN classification based on the final prediction probability and the ensemble logic. Specifically, if the prediction probability is greater than the ensemble threshold, it is predicted as IgAN; otherwise, it is predicted as non-IgAN.

[0064] In another specific embodiment, Shap analysis enables model interpretability to ensure clinical reliability. Shapley additive interpretation (Shap) performs post-hoc interpretability verification, calculating clinical features for each tabular branch. i Importance score The importance score is calculated as follows; and an attention heatmap is used to display areas of diagnostic significance.

[0065] Example 3: The training method for the prediction model includes the following steps: Step 201: Data acquisition and preprocessing.

[0066] Facial and tongue images were acquired using a Traditional Chinese Medicine Tongue and Face Imaging Instrument (model: SMF-S1), which is equipped with a uniform fixed LED light source and a Canon camera, with an image resolution of 1728 × 2592 pixels.

[0067] The tongue region in facial photographs is segmented using the zero-shot Segment Anything Model (SAM) method, reducing the need for manual annotation. The segmented image undergoes center cropping and bilinear scaling, resulting in a final size of 224. 224 pixels, but not limited to this.

[0068] Facial images are preprocessed using center cropping to focus on capturing areas of diagnostic value.

[0069] Clinical characteristics: Collect demographic characteristics (sex, age, height, weight, BMI), medical history (hypertension, hyperuricemia, diabetes, cardiovascular disease, tumor, hyperlipidemia), and chief complaint (initial symptoms and course of disease).

[0070] Laboratory indicators: complete blood count (white blood cell count, hemoglobin), urine tests (quantitative urine red blood cells, quantitative 24-hour urine protein, urine volume, urine protein concentration), biochemical indicators (total protein, albumin, alanine aminotransferase, aspartate aminotransferase, alkaline phosphatase, blood urea nitrogen, creatinine, estimated glomerular filtration rate (eGFR), serum uric acid, total cholesterol, triglycerides, creatine kinase, low-density lipoprotein), immune and inflammatory indicators (C-reactive protein, complement C3 and C4, immunoglobulin IgA / IgE / IgG / IgM, IgA / C3 ratio, autoimmune antibody profile), phospholipase A2 receptor antibody, infection and immune screening items (pre-transfusion eight items, anti-neutrophil cytoplasmic antibody, glomerular basement membrane antibody), and a complete thyroid function panel.

[0071] In the specific training, the clinical data included 46 indicators. Since not all patients underwent all tests in real-world clinical applications, common missing values ​​were encoded with -1 to preserve their informational value. This differs from traditional imputation methods, which often mask clinically significant missing value patterns. All features were standardized using z-scores with cohort-specific statistics.

[0072] The model training set included 669 cases, the test set included 148 cases, the follow-up set included 162 cases during treatment, and the prospective set included 108 cases. The prospective set cohort did not include images or severe cases.

[0073] Step 202: Model Training. Specifically, weighted multi-task learning is used for end-to-end training.

[0074] (15); Tongue Indicated as the top expert, face Indicated as the second expert, tabular Indicated as a third expert, fusion Indicated as the fourth expert, This is represented as the loss in ensemble learning. Let represent the loss of the b-th expert. The learning rate for each branch is: , , , image This includes tongue and facial branches. The optimization process uses the AdamW algorithm with a cosine annealing strategy, lasting for 100 training epochs. Gradient clipping. Ensure training stability.

[0075] The training was conducted on an NVIDIA RTX 4090 GPU with 24GB of VRAM. The framework was implemented using PyTorchLightning 2.0, employing deterministic computation (seed = 2025) to ensure reproducibility. Mixed-precision computing (FP16) was used for training to optimize memory usage and computational efficiency. The batch size was set to 16, with an effective batch size of 32 achieved through two-step gradient accumulation. Data loading utilized four worker processes with persistent memory to minimize I / O overhead. All experiments were performed using CUDA 11.8 with the cuDNN 8.9 accelerator for deep learning computation.

[0076] Stratified five-fold cross-validation was used to segment patients into models; the AUROC difference between models was assessed using the Delon test, and classification consistency was assessed using the McNimar test; all p-values ​​were adjusted using Bonferroni correction. The confidence intervals were calculated using a bootstrap resampling method with 1000 iterations. To assess the contribution of different modalities, the performance of each branch was evaluated independently.

[0077] like Figure 12 In the test set (n = 148): Single-modal performance in the model: AUC for the table branch was 0.979, AUC for the face branch was 0.836, AUC for the tongue branch was 0.873, and AUC for the fusion branch was 0.991. Multimodal performance in the model: Clinical data + facial images combined (AUC = 0.981); Clinical data + tongue images combined (AUC = 0.988); Clinical data + facial images + tongue images combined (AUC = 0.998, SEN = 0.980, SPE = 1.000). Figure 13 In the follow-up set, the prediction model's AUC was 0.979, SEN was 0.877, and SPE was 0.810. Figure 14 In the scenario where image modalities are missing, the AUC of the prediction model on the lookahead set is 0.943, a decrease of 5.5%.

[0078] In the test set, the predictive model demonstrated excellent non-invasive diagnostic performance for IgAN: AUC of 0.998 (95% CI: 0.998-0.998), accuracy of 0.980 (95% CI: 0.978, 0.980), sensitivity of 1.000 (95% CI: 0.998, 1.000), specificity of 0.966 (95% CI: 0.964, 0.976), and F1 score of 0.976 (95% CI: 0.975, 0.976).

[0079] Under complete multimodal input conditions (clinical data, tongue and facial images), a systematic comparison was conducted with current mainstream multimodal methods (including Mult+XGB, MCAN+XGB, MMBT+XGB, and TIP+XGB). In the internal test set, the IgAN-Seeker predictive model performed best on all evaluation metrics, significantly outperforming all baseline models. Specifically, IgAN-Seeker achieved an AUC of 0.998 (0.982) and an F1 score of 0.976 (0.958), demonstrating superior IgAN prediction compared to other models.

[0080] The tabular branch was compared and evaluated against several advanced tabular data processing models. The tabular branch achieved an AUC of 0.979; in contrast, the FT-Transformer had an AUC of 0.902, ranking second. Other CNN models (such as TabularAttnMode, XgboostLikeNN, and NPT) showed significant performance gaps, with AUCs all below 0.9, and all other metrics (ACC, SEN, SPE, F1) failing to exceed 0.95. These performance comparisons demonstrate that the tabular branch exhibits a significant advantage in utilizing clinical data for IgAN diagnosis.

[0081] In the evaluation of the facial image branch, IgAN-Seeker achieved an AUC of 0.836, outperforming other comparative models such as ShuffleNet_v2_x0_5 (AUC 0.829), ResNet34 (AUC 0.824), and ConvNext_base (AUC 0.818). Furthermore, IgAN-Seeker demonstrated significant advantages in accuracy (0.804), sensitivity (0.738), specificity (0.874), and F1 score (0.769).

[0082] For the tongue image branch, IgAN-Seeker achieved an AUC of 0.873, outperforming other models such as ConvNext_base (AUC = 0.820), DenseNet121 (AUC = 0.804), and ViT_B_16 (AUC = 0.786). These results further demonstrate the potential of IgAN-Seeker in non-invasive diagnosis of IgAN.

[0083] To evaluate the diagnostic efficacy of IgAN-Seeker in real-world clinical scenarios, the predictive model was compared with the performance of physicians at various levels (junior, intermediate, associate senior, and senior) on the test set. The average diagnostic performance of the physician group was as follows: accuracy 0.809, sensitivity 0.701, specificity 0.885, and F1 score 0.755. In contrast, IgAN-Seeker achieved a significant improvement in diagnostic performance, with its accuracy, sensitivity, specificity, and F1 score increasing by 21.1%, 40.8%, 10.2%, and 29.3% respectively compared to the physician group average.

[0084] Table 1 shows the diagnostic capabilities of experts and predictive models on the test and prospective sets.

[0085] Table 1

[0086] Example 4 provides an IgAN prediction system based on multimodal fusion, such as... Figure 12 As shown, it includes an input module, a sixth prediction module, and an output module. The input module is used to obtain the data to be predicted; The sixth prediction module is used to predict the data to be predicted using a prediction model based on multimodal fusion, and obtain the fifth prediction result. The output module is used to output the fifth prediction result.

[0087] Example 5 provides an IgAN prediction device, including a memory and a processor. The memory stores code, and when the code is executed by the processor, it implements the above-described IgAN prediction method.

[0088] This invention presents a non-invasive prediction method based on a multimodal deep learning framework, integrating three complementary data sources: clinical data, facial features, and tongue image features. It employs four specialized branches—a table branch, a facial branch, a tongue branch, and a fusion branch—and achieves integration through an expert hybrid integration mechanism. It maintains robust performance even when some modal data is missing, and a dynamic routing mechanism adaptively selects available experts for prediction. It addresses a key challenge in multimodal medical data fusion: effectively integrating heterogeneous modal data—clinical measurement data, facial features, and tongue image visual texture—while maintaining the independent contributions of each modality. Through complementary information extraction and a hierarchical fusion strategy, it achieves superior diagnostic accuracy.

[0089] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An IgAN prediction method based on multimodal fusion, characterized in that, Includes the following steps: Collect data to be predicted; The fifth prediction result is obtained by using a prediction model based on multimodal fusion to predict the data to be predicted; The prediction model's network includes the following branches: tongue image branch, face branch, table branch, fusion branch, gating module, and fifth prediction module. The tongue image branch is used to extract tongue image features from the tongue image image; And by having the first expert predict the tongue image, the first prediction result is obtained; The facial branch is used to extract facial features from a facial image and to predict the facial image by a second expert to obtain a second prediction result; The table branch is used to extract table features from clinical data based on graph neural networks, and to obtain a third prediction result by using a third expert to predict the clinical data. The fusion branch is used to fuse tongue features, facial features, and table features to obtain a third fusion feature, and a fourth expert predicts the third fusion feature to obtain a fourth prediction result. The gating module is used to select important experts from the first expert, the second expert, the third expert, and the fourth expert; The fifth prediction module is used to obtain a fifth prediction result based on the prediction results of the key experts.

2. The IgAN prediction method according to claim 1, characterized in that, The facial branch includes a second feature extraction module, a clustering graph attention module, and a second prediction module; The facial image is input into the second feature extraction module, which is connected to the aggregation map attention module. The output values ​​of the second feature extraction module and the aggregation graph attention module are concatenated and then pooled to obtain the second concatenated feature; The second concatenated feature is then subjected to global average pooling to obtain the second classification feature; On the one hand, the second classification feature is predicted by the second prediction module; on the other hand, facial features are obtained after the second classification feature is projected.

3. The IgAN prediction method according to claim 1, characterized in that, The tongue branch includes a first feature extraction module, a Transformer module, and a first prediction module; The tongue image is processed by the first feature extraction module and the Transformer module to obtain the first classification features; On the one hand, the first classification feature is predicted by the first prediction module to obtain the first prediction result; On the other hand, after the first classification features pass through the first projection layer, tongue image features are obtained. ； The first feature extraction module includes a fifth convolutional layer, a first spatial attention submodule, a sixth convolutional layer, a second spatial attention submodule, a seventh convolutional layer, a third spatial attention submodule, an eighth convolutional layer, and a pooling operation connected in sequence. The spatial attention submodule includes: concatenating the first feature map Fl after average pooling and max pooling respectively, passing it through a convolutional layer and activation function, and concatenating the resulting feature map with the first feature map by dot product to obtain the second feature map.

4. The IgAN prediction method according to claim 1, characterized in that, The table branches include a graph neural network submodule, a gradient boosting submodule based on the XGBoost method, and a self-attention submodule; The graph neural network submodule is sequentially connected to the self-attention submodule and the linear layer to obtain the fourth feature; The XGB features extracted by the gradient boosting submodule are concatenated with the fourth feature after passing through the projection layer, and then passed through the fourth projection layer to obtain the table features. The table features are predicted by the third prediction module to obtain the third prediction result.

5. The IgAN prediction method according to claim 1, characterized in that, The table branches include a graph neural network submodule, a gradient boosting submodule based on the XGBoost method, and a self-attention submodule; The tabular data is connected to the self-attention submodule; On the one hand, the output of the self-attention submodule is connected to the graph neural network submodule to obtain the fourth feature; On the other hand, after the output of the attention submodule is connected to the gradient boosting submodule and the third projection layer, it is concatenated with the fourth feature, and then passed through the linear layer and the fourth projection layer to obtain the table feature; The table features are predicted by the third prediction module to obtain the third prediction result; Among them, table features h tab Represented as: ； in, NTP() For linear layer calculation, GCN() Graph neural network computation; MLP() Calculations for the projection layer, L XGB features extracted for the gradient boosting submodule SELU() Calculations are performed for the fourth projection layer; This is the output value of the self-attention submodule.

6. The IgAN prediction method according to claim 1, characterized in that, The fusion branch includes a first fusion submodule and a second fusion submodule. The first fusion submodule is used to fuse two sub-features to obtain a fused sub-feature, wherein the sub-feature is selected from tongue image features, table features and facial features; The second fusion submodule is used to fuse the first fusion feature and the second fusion feature to obtain the third fusion feature; The fourth prediction module is used to predict the third fusion feature to obtain a fourth prediction result.

7. The IgAN prediction method according to claim 6, characterized in that, The first fusion submodule projects the sub-features onto the projection space, which includes a coarse-level projection layer, a medium-level projection layer, and a fine-level projection layer. The fourth concatenated feature is obtained by multiplying the projected features of the corresponding projection layers of the two sub-features. After upsampling the fourth concatenation feature, a power set concatenation operation is performed, and the fifth concatenation feature is obtained through a linear layer. The precise projection features of the two sub-features are stitched together, and then fused with the fifth stitched feature after passing through the Transformer module to obtain the fused sub-feature; The two sub-features include tongue image features and table features, and the fusion sub-features include the first fusion feature; Alternatively, the two sub-features include facial features and table features, and the fused sub-features include a second fused feature.

8. The IgAN prediction method according to claim 1, characterized in that, The probability of the fifth prediction result is expressed as follows: ； in, x This is represented as the input sample. h concat This is a concatenated vector obtained by encoding the various modalities of the input sample separately. W gate Represented as gating weight, g The importance score vector for experts; top-2(g) This indicates that the two experts with the highest scores will be selected; g top-2 This represents the importance score vector of the two experts with the highest scores. softmax() For normalization operations, p k This represents the prediction probability of the k-th expert; This represents the probability of the fifth prediction result.

9. A prediction system, characterized in that, For implementing the IgAN prediction method as described in any one of claims 1-8, the prediction system includes an input module and a sixth prediction module. The input module is used to obtain the data to be predicted; The sixth prediction module is used to predict the data to be predicted using a prediction model based on multimodal fusion, and obtain the fifth prediction result.

10. A prediction device, characterized in that, It includes a memory that stores code, which, when executed, implements the IgAN prediction method as described in any one of claims 1-8.

Images