A Diagnostic Aid Method and System Based on Multimodal Decoupling Dynamic Graph Learning

Abstract

This invention relates to the field of intelligent brain disease diagnosis technology, specifically providing an auxiliary diagnostic method and system based on multimodal decoupled dynamic graph learning. The method includes: acquiring and preprocessing multimodal data (such as neuroimaging, genetic markers, etc.) of the subject; extracting common pathological information and modality-specific features through a shared encoder and modality-specific encoders respectively, and optimizing the separation process using a decoupling loss function; furthermore, fusing all modality embeddings using a multi-head self-attention mechanism with a masked matrix to generate initial node representations, where the mask is used to suppress modality self-attention; subsequently, performing hierarchical dynamic graph convolution based on the node representations: in each layer, dynamically updating the graph adjacency matrix by combining the current node representation with the original features, and iteratively optimizing the node representations through message passing; finally, inputting the optimized representations into a classifier to obtain disease prediction results. This invention improves the automation performance and reliability of diagnosis.

Description

Technical Field

[0001] This invention belongs to the field of intelligent brain disease diagnosis technology, specifically relating to an auxiliary diagnostic method and system based on multimodal decoupling dynamic graph learning. Background Technology

[0002] In the current field of intelligent brain disease diagnosis, the integration of multimodal data, including neuroimaging, genetic markers, and clinical assessments, has become a key approach to improve diagnostic performance. Graph learning-based methods, by modeling the similarity relationships between subjects, can effectively utilize group-level information to assist individual diagnosis, making them a research hotspot in this area. However, existing solutions still have significant limitations: First, in the multimodal fusion stage, traditional feature-level or decision-level fusion strategies struggle to effectively decouple cross-modal shared information from modality-specific noise, resulting in the suppression of effective information in low-quality modalities. Second, in terms of graph construction and reasoning, most methods rely on static, fixed graph structures, failing to depict the complex and dynamically evolving relationships between subjects during disease progression, and often overemphasize inter-class separation, neglecting the commonalities of real pathology. These problems limit the in-depth exploration of the collaborative value of multimodal data and also affect the accuracy and clinical interpretability of diagnosis. Therefore, there is an urgent need for an end-to-end learning framework capable of modality decoupling and dynamic relationship modeling to more accurately and robustly support the auxiliary diagnosis of brain diseases. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention provides an auxiliary diagnostic method and system based on multimodal decoupling dynamic graph learning, in order to solve the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides an auxiliary diagnostic method based on multimodal decoupling dynamic graph learning, comprising:

[0005] Multimodal data from multiple subjects were acquired and preprocessed; the multimodal data included at least two of the following: neuroimaging, genetic markers, and clinical assessment data.

[0006] For each subject, shared embeddings are extracted from each modality's data using a shared encoder shared across all modalities, and modality-specific embeddings are extracted using a modality-specific encoder corresponding to each modality. The extraction process is optimized using a decoupling loss function to separate common pathological information across modalities from unique feature information of each modality.

[0007] For each subject, a multi-head self-attention mechanism with an introduced mask matrix is ​​used to fuse the shared embeddings and modality-specific embeddings of all their modalities to generate the subject's initial node representation; the mask matrix is ​​used to suppress modality attention to itself;

[0008] Based on the initial node representations of all subjects, hierarchical dynamic graph convolution is performed to iteratively update the node representations, resulting in optimized node representations. In each layer, the graph adjacency matrix is ​​dynamically calculated and updated based on the current node representations and the original multimodal features, and message passing and feature aggregation between nodes are performed based on the updated adjacency matrix.

[0009] The optimized node representations of each subject are input into the classifier to obtain the prediction results of the disease category.

[0010] In an optional implementation, multimodal data from multiple subjects is acquired and preprocessed, including:

[0011] Multimodal raw data from multiple subjects are obtained from a pre-set medical database; the multimodal raw data includes at least neuroimaging data, genetic marker data, and clinical assessment data.

[0012] The multimodal raw data is preprocessed, including: standardizing the multimodal raw data to eliminate dimensional differences between different modal data; filling in modal data with missing values; and removing outliers and noisy data.

[0013] In an optional implementation, shared embeddings are extracted from the modal data using a shared encoder shared by all modalities, including:

[0014] For the original feature matrix of the m-th modality, a linear projection layer is used to map the original feature matrix to a common feature space of dimension D to obtain standardized intermediate features;

[0015] The standardized intermediate features of each modality are input into a shared encoder shared by all modalities. The shared encoder is a multilayer perceptron (MLP). Through its parameter sharing mechanism, cross-modal common features are extracted, and the output is a shared embedding for each modality.

[0016] In one optional implementation, modality-specific embeddings are extracted using modality-specific encoders corresponding to each modality, including: for the standardized intermediate features of the m-th modality, inputting them into a modality-specific encoder dedicated to the m-th modality; the modality-specific encoder is a multilayer perceptron (MLP) independently configured for the data characteristics of that modality, used to extract and output feature information unique to that modality, to obtain the modality-specific embedding of the m-th modality.

[0017] In an optional implementation, the method further includes a decoupling loss function, which comprises a cross-orthogonal loss term, a cross-modal decoupling loss term, and a total decoupling loss term obtained by a weighted sum of the two; wherein:

[0018] The cross-orthogonal loss term reduces the linear correlation between shared embeddings and modality-specific embeddings of the same modality by calculating and minimizing the norm of their cross-correlation matrix.

[0019] The cross-modal decoupling loss term is calculated for shared embeddings and modality-specific embeddings respectively. It maintains the information of each modality by penalizing the deviation of the diagonal elements and the unit value in the cross-modal correlation matrix, and at the same time penalizes the off-diagonal elements to suppress redundant associations of the same type of embeddings between different modalities.

[0020] The total decoupling loss term is used to collaboratively optimize the cross-orthogonal loss term and the cross-modal decoupling loss term during model training, so as to drive the separation of shared embedding and modality-specific embedding.

[0021] In an optional implementation, for each subject, a multi-head self-attention mechanism incorporating a mask matrix is ​​used to fuse the shared embeddings and modality-specific embeddings from all their modalities to generate an initial node representation for that subject, including:

[0022] For the i-th subject, the shared embeddings of all their modalities constitute a shared embedding sequence. And the modality-specific embeddings of all its modalities constitute a specific embedding sequence. , where M is the total number of modes;

[0023] The shared embedding sequence and the specific embedding sequence are input into a multi-head self-attention module, which generates a query matrix Q, a key matrix K, and a value matrix V through linear projection, respectively.

[0024] In the attention weight calculation, a mask matrix of size M×M is introduced, with the diagonal elements of the matrix set to negative infinity, to suppress the attention of each modality to itself;

[0025] general formula Calculate the attention output, where The dimension of the attention head;

[0026] The sequence output by the attention module is input into a multilayer perceptron (MLP) for nonlinear fusion and feature compression, and the initial node representation of the subject is output.

[0027] In an optional implementation, the sequence output by the attention module is input into a multilayer perceptron (MLP) for nonlinear fusion and feature compression, outputting the subject's initial node representation, including:

[0028] The optimized shared embedding sequence output by the attention module and the specific embedding sequence are concatenated along the feature dimension to obtain the concatenated feature.

[0029] The spliced ​​features are input into the multilayer perceptron (MLP); the MLP performs nonlinear transformation and dimensionality reduction on the spliced ​​features and outputs the initial node representation of the subject.

[0030] In one optional implementation, based on the initial node representations of all subjects, hierarchical dynamic graph convolution is performed to iteratively update the node representations, resulting in optimized node representations, including:

[0031] For the l-th layer (l=0,1,...,L-1), the following iterative update process is performed, where the node representation of the 0-th layer is the initial representation of the node:

[0032] The node representations of layer l are used as input and projected into query vector, key vector and value vector respectively. After normalizing the query vector and key vector, the dynamic attention output of this layer is obtained through an attention calculation function that integrates global node information as residuals.

[0033] Based on the original multimodal features of all subjects, the feature similarity between each pair of subjects is calculated; the calculated similarity matrix is ​​subjected to non-negative activation and sparsification processing to retain the connection between each subject and the most similar preset number of other subjects; the processed matrix is ​​normalized to generate the dynamic adjacency matrix of the l-th layer.

[0034] The intermediate update representation is obtained by fusing the dynamic attention output with the neighbor information of each node aggregated through the dynamic adjacency matrix; the intermediate update representation is then weighted and fused with the node representation of the l-th layer through residual connection, and the fusion result is subjected to layer normalization processing to finally output the node representation of the (l+1)-th layer.

[0035] The iteration continues until the preset total number of graph convolutional layers L is reached. The node representation of the Lth layer output in the last iteration is used as the optimized node representation.

[0036] In an optional implementation, the optimized node representations of each subject are input into a classifier to obtain a prediction result for the disease category, including:

[0037] The optimized node representation of each subject is input into a classification network. The classification network maps the node representation to the same dimension as the number of disease categories through a fully connected layer, and then processes it through a Softmax function to output the predicted probability of the subject belonging to each disease category.

[0038] During the model training phase, the predicted probability is optimized using a class-weighted cross-entropy loss function, where the weight of each class is inversely proportional to the number of samples of that class in the training data, in order to alleviate the class imbalance problem.

[0039] Secondly, the present invention provides an auxiliary diagnostic system based on multimodal decoupling dynamic graph learning, comprising:

[0040] The data acquisition module is used to acquire and preprocess multimodal data from multiple subjects; the multimodal data includes at least two of the following: neuroimaging, genetic markers, and clinical assessment data.

[0041] The modality decoupling module is used to extract shared embeddings from each modality's data for each subject using a shared encoder shared by all modalities, and to extract modality-specific embeddings using a modality-specific encoder corresponding to each modality. The extraction process is optimized by a decoupling loss function to separate common pathological information across modalities from feature information unique to each modality.

[0042] The modality fusion module is used to generate an initial node representation for each subject by fusing the shared embeddings and modality-specific embeddings of all its modalities using a multi-head self-attention mechanism that introduces a mask matrix; the mask matrix is ​​used to suppress modality attention to itself.

[0043] The dynamic transmission module is used to perform hierarchical dynamic graph convolution to iteratively update the node representations based on the initial node representations of all subjects, so as to obtain optimized node representations. In each layer, the graph adjacency matrix is ​​dynamically calculated and updated based on the current node representation and the original multimodal features, and message passing and feature aggregation between nodes are performed based on the updated adjacency matrix.

[0044] The diagnostic prediction module is used to input the optimized node representations of each subject into the classifier to obtain the prediction results of the disease category.

[0045] The beneficial effects of this invention lie in the fact that the auxiliary diagnostic method and system based on multimodal decoupling dynamic graph learning provided by this invention proposes a systematic innovation to address the shortcomings of existing multimodal graph learning methods, demonstrating significant beneficial effects in practice. First, through a multimodal decoupling mechanism and a dedicated loss function, the model can clearly separate shared pathological information from modality-specific features, effectively activating and utilizing low-quality modality data that is easily overlooked in traditional methods, significantly enhancing the robustness of information utilization. Second, the dynamic graph learning module breaks free from the limitations of fixed topology structures, enabling it to adaptively characterize the complex and evolving relationships between subjects with varying network depths. Its modeling approach better reflects the clinical reality of the continuity of disease progression and the commonalities in pathology. Experiments on multiple publicly available brain disease datasets (such as ADNI and ABIDE) show that this method consistently outperforms existing state-of-the-art methods such as MMGL and HMFGL in diagnostic accuracy, sensitivity, and specificity in multi-class and binary classification tasks. Especially in scenarios with limited training data (e.g., only 10% labeled samples), it still maintains excellent generalization performance, demonstrating its strong clinical applicability potential. Furthermore, the decoupled representations and dynamic relationship patterns learned by the model can be used for interpretability analysis (such as SHAP) to reveal key biomarkers that highly agree with known pathological mechanisms and their cross-modal associations, providing valuable insights for disease subtype stratification and mechanism research. In summary, this approach not only improves the automation performance and reliability of assisted diagnosis but also provides new analytical tools for understanding the disease itself. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0048] Figure 2 This is another illustrative flowchart of a method according to an embodiment of the present invention.

[0049] Figure 3 This is a schematic block diagram of a system according to an embodiment of the present invention. Detailed Implementation

[0050] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0052] The method provided in this embodiment of the invention is executed by a computer device, and correspondingly, the auxiliary diagnostic system based on multimodal decoupling dynamic graph learning runs in the computer device.

[0053] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The implementing entity can be an auxiliary diagnostic system based on multimodal decoupling dynamic graph learning. Depending on different requirements, the order of steps in this flowchart can be changed, and some can be omitted.

[0054] like Figure 1 As shown, the method includes:

[0055] S1. Acquire multimodal data from multiple subjects and preprocess it; the multimodal data includes at least two of the following: neuroimaging, genetic markers, and clinical assessment data;

[0056] S2. For each subject, shared embeddings are extracted from each modality data using a shared encoder shared by all modalities, and modality-specific embeddings are extracted using a modality-specific encoder corresponding to each modality. The extraction process is optimized by a decoupling loss function to separate common pathological information across modalities from feature information unique to each modality.

[0057] S3. For each subject, a multi-head self-attention mechanism with an introduced mask matrix is ​​used to fuse the shared embeddings and modality-specific embeddings of all its modalities to generate the subject's initial node representation; the mask matrix is ​​used to suppress modality attention to itself;

[0058] S4. Based on the initial node representations of all subjects, perform hierarchical dynamic graph convolution to iteratively update the node representations and obtain optimized node representations; wherein, in each layer, the graph adjacency matrix is ​​dynamically calculated and updated according to the current node representations and the original multimodal features, and message passing and feature aggregation between nodes are performed based on the updated adjacency matrix.

[0059] S5. Input the optimized node representations of each subject into the classifier to obtain the prediction results of the disease category.

[0060] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0061] S101. Multimodal Data Acquisition

[0062] Multimodal raw data from multiple subjects were collected from authoritative medical research databases or clinical institutions. The data contained at least two or all of the following three types:

[0063] Neuroimaging data: such as structural magnetic resonance imaging (sMRI), functional magnetic resonance imaging (fMRI), or positron emission tomography (PET) image data, which need to have undergone preliminary imaging processing (such as registration, standardization, segmentation) and have feature vectors or atlas data of regions of interest (ROI) extracted.

[0064] Genetic marker data: such as single nucleotide polymorphism (SNP) data, gene expression data, or epigenetic data, are usually represented in vector or matrix form.

[0065] Clinical assessment data includes neuropsychological scale scores (such as MMSE, ADAS-Cog), cognitive test results, demographic information (such as age, gender, years of education) and other behavioral indicators.

[0066] S102. Data Preprocessing Flow

[0067] The acquired raw data undergoes systematic cleaning and standardization, specifically including:

[0068] Standardization: Z-score standardization or min-max normalization is performed on the features of each modality to eliminate the bias caused by differences in units and magnitudes between different features, and to ensure that all features are within a comparable numerical range.

[0069] Missing value handling: For missing values ​​in the data, appropriate methods such as mean / median imputation, KNN imputation, or model prediction imputation are used to fill in missing values ​​according to the missing pattern and data characteristics, so as to ensure the integrity of the dataset.

[0070] Outlier and noise removal: Use statistical methods (such as based on standard deviation or interquartile range) or visualization methods to identify and remove outlier samples or noise features that deviate significantly from the normal distribution to prevent them from interfering with model training.

[0071] Feature alignment and formatting: Ensure that data from different modalities from the same subject strictly correspond in sample labeling. Finally, organize the data for each modality into a matrix format. Where N is the number of subjects, D m Let be the feature dimension of the m-th modality.

[0072] S103. Dataset Partitioning and Management

[0073] The preprocessed complete multimodal dataset is randomly and hierarchically divided into three independent subsets according to a preset ratio (e.g., 70% training set, 15% validation set, and 15% test set) to maintain the proportion of each category. These subsets are used for model parameter training, hyperparameter tuning, and final performance evaluation, respectively.

[0074] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0075] A "linear projection + dual encoder" architecture is adopted to achieve accurate mapping from raw modal features to two types of embeddings. The specific process is as follows:

[0076] Modal feature unification mapping: First, each original modal feature matrix is ​​mapped through a linear projection layer. (in For the number of subjects, For the first The original feature dimensions of each modality (Total number of modalities) mapped to dimension From the common feature space, we obtain the standardized intermediate features. This ensures that features of different modalities are comparable and have a basis for interaction.

[0077] Dual encoders generate embeddings in parallel:

[0078] Shared encoder: Employs a multilayer perceptron (MLP) structure shared across all modalities. Through parameter sharing, it forces the learning of common features applicable across modalities, outputting the first... Shared embedding of each modality ,in This embedding focuses on consistent pathological associations across different modalities (such as brain function decline features reflected in both neuroimaging and clinical indicators).

[0079] Modality-specific encoders: Each modality is configured with a dedicated MLP network, performing customized feature extraction based on the data source characteristics of each modality (such as structural features of MRI and sequence features of genetic markers), and outputting specific embeddings. ,in The embedding preserves disease characteristics unique to this modality that cannot be replaced by other modalities (such as structural abnormalities in specific brain regions in MRI and behavioral-specific indicators in clinical assessment).

[0080] To ensure effective separation of shared and specific embeddings, and consistency and independence of cross-modal features, a three-level loss function is designed for collaborative optimization, as follows:

[0081] Cross-orthogonal loss ( The core objective is to minimize the correlation between shared and specific embeddings within the same modality, thereby reducing feature redundancy. This is achieved by calculating the cross-correlation matrix between the two types of embeddings. ( The overall correlation of the matrix is ​​measured by the Frobenius norm, and the loss function is defined as follows:

[0082]

[0083] This loss enhances complementarity by penalizing the linear correlation between the two types of embeddings, forcing them to focus on feature information in different dimensions.

[0084] Cross-modal decoupling loss ( ): Designed separately for shared embeddings and specific embeddings, ensuring consistency (information preservation) of embeddings of the same type across modalities and independence (decorrelation) between different modalities. First, the embeddings are... Normalization ( , ), calculate the cross-modal correlation matrix (Specific embedding is obtained similarly) The loss function is defined as:

[0085]

[0086] in and To balance hyperparameters, For the embedding dimension, the first term penalizes the deviation of the diagonal elements from 1, ensuring that embeddings of the same type in each modality retain their core information; the second term penalizes the sum of squares of the off-diagonal elements, suppressing redundant associations between embeddings of the same type in different modalities, and achieving efficient separation of cross-modal features.

[0087] Total decoupling loss: Integrating the above losses through hyperparameters By controlling the strength of the cross-orthogonal constraints, the final loss function is:

[0088]

[0089] and Cross-modal decoupling losses for shared embedding and specific embedding, respectively. The superscript 's' represents shared embeddings, and the superscript 'spe' represents specific embeddings. Through collaborative optimization, a decoupling effect is achieved where "embeddings of the same type are consistent across modalities, while embeddings of different types are independent across modalities."

[0090] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0091] For each subject, a multi-head self-attention mechanism with an introduced mask matrix is ​​used to fuse the shared embeddings and modality-specific embeddings of all their modalities to generate the subject's initial node representation; the mask matrix is ​​used to suppress modality attention to itself.

[0092] S301. Inputs and Core Objectives

[0093] Input for each subject Two types of embedded sequences: shared embedded sequences ( For the first In the mode, the th Shared embeddings of individual subjects and specific embedding sequences The core objective is to overcome the problems of insufficient cross-modal interaction or noise interference in traditional fusion methods. By adaptively capturing complementary cross-modal information through an attention mechanism, while avoiding information redundancy caused by modal self-attention, a unified and high-quality initial representation of subject-level nodes is ultimately generated.

[0094] S302. Multi-head self-attention fusion mechanism with masking

[0095] A Transformer-based multi-head self-attention architecture is adopted, and a customized mask matrix is ​​introduced to achieve accurate cross-modal fusion:

[0096] Mask matrix design: Construct a matrix with a size of mask matrix The diagonal elements are set to The remaining positions are set to 0. The core purpose of this design is to prohibit each modal feature from assigning attention weights to itself (i.e., to avoid "self-attention"), and to force the model to focus on the interaction information between different modalities, thereby maximizing the mining of cross-modal complementarity.

[0097] Attention calculation process:

[0098] Feature projection: sharing embedded sequences and specific embedding sequences Projected into query (Q), key (K), and value (V) matrices respectively, i.e. , , ,in ( For the number of attention heads, For each dimension of the attention head, satisfying ).

[0099] Attention weight calculation: Weights are calculated using the dot product attention formula, and a mask matrix is ​​added to suppress self-attention. The formula is as follows:

[0100]

[0101] in This is a scaling factor used to mitigate the gradient vanishing problem caused by dot product operations; The function ensures that the sum of attention weights is 1, thus achieving adaptive weight allocation.

[0102] Multi-head integration: The outputs of each attention head are concatenated and integrated into a unified feature through a linear projection layer, resulting in an attention-optimized shared embedding sequence. and specific embedding sequences .

[0103] S303. Subject-level node representation generation

[0104] A multilayer perceptron (MLP) is used to finally fuse the two types of optimized embedding sequences, specifically: and Concatenated according to feature dimensions The input is processed by an MLP network for nonlinear transformation and feature compression, and the output is the subject. Initial representation of nodes ( (This is the node representation dimension). This representation integrates cross-modal common information with modal-specific information, providing high-quality node input for subsequent dynamic graph modeling.

[0105] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0106] S401. Core Objectives and Design Philosophy

[0107] To address the limitations of existing methods that static graph structures cannot capture the dynamic and multifaceted relationships among subjects, this module adopts "hierarchical dynamic updates" as its core design concept to achieve two main goals: first, to adaptively adjust the similarity weights among subjects based on the dynamic changes in node representations, thereby constructing a dynamically evolving graph structure; and second, to achieve efficient propagation and aggregation of node information through hierarchical message passing, while preserving global information and local connections, so that the final node representation can accurately reflect the complex relationships among subjects during disease progression (such as the pathological commonalities of different disease stages).

[0108] S402. Dynamic Attention Calculation

[0109] The Simple Kernel Attention mechanism is adopted, which combines the global residual term and the stabilization factor to realize the dynamic adaptive calculation of node attention weights. The specific process is as follows:

[0110] Feature projection and normalization: representing the initial nodes of all subjects The projection is a Q, K, V tensor, i.e. , , ,in ( For dynamic attention heads, (dimensions for each head), to obtain Then Q and K were analyzed. Normalization ensures the stability of attention calculations.

[0111] Dynamic attention output: Introducing a global residual term (Preserving global node information) and stability factor (To avoid numerical fluctuations caused by an excessively small denominator), the attention output formula is:

[0112]

[0113] in For a vector of all 1s, this formula combines local attention weights ( It enables dynamic and robust interaction of node features with global information.

[0114] S403. Construction of Dynamic Adjacency Matrix

[0115] Instead of using the traditional fixed adjacency matrix design, we dynamically construct an adaptive adjacency matrix based on the original features and node representations.

[0116] Similarity calculation: based on the original multimodal feature matrix (Concatenated by feature dimension), calculate subjects and cosine similarity To capture subject similarity at the level of original features.

[0117] Sparsity and Normalization: Eliminating Negative Similarity Using the ReLU Function ( The top-k most similar nodes for each subject are retained (k is a hyperparameter controlling the sparsity of the graph), resulting in a sparse similarity matrix; this matrix is ​​then symmetrically normalized. Finally, the dynamic adjacency matrix is ​​obtained. This matrix adaptively adjusts as the original features and node representations change, enabling it to capture the dynamic relationships between subjects.

[0118] S404. Hierarchical Message Passing and Node Representation Update

[0119] A hierarchical iterative message passing mechanism, combined with dynamic attention output and adjacency matrix, is used to progressively optimize node representations. The specific process is as follows:

[0120] Intra-layer message aggregation: Layer node embedding Obtained through dynamic attention calculation At the same time, through the adjacency matrix Aggregate local neighbor information ( The two are then fused element-wise to obtain an intermediate characterization:

[0121]

[0122] Residual Connections and Layer Normalization: To alleviate gradient vanishing and improve training stability, residual connections and layer normalization (LN) are introduced. The update formula is as follows:

[0123]

[0124] in LN is the residual balance coefficient, used to adjust the weight ratio between new aggregated information and historical representations; LN normalizes the feature dimensions, accelerating model convergence.

[0125] Multi-round iterative optimization: Repeat the above process Second-rate( (The number of convolutional layers in the graph is adaptively adjusted according to the task), ultimately yielding the final result. Node representation optimized by layered dynamic message passing This representation fully integrates global information, local neighbor associations, and dynamic relationship features, providing a highly discriminative input for diagnosis and prediction.

[0126] In one embodiment of the present invention, based on step S5, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0127] Based on final node representation The system maps to the class probability space through a fully connected layer and uses a class-weighted cross-entropy loss function to optimize diagnostic prediction performance, as detailed below:

[0128] Category probability output: ,in ( (Number of disease categories) For bias terms, For the first The probability distribution of categories for each subject.

[0129] Class-weighted cross-entropy loss: To address the potential class imbalance problem in clinical data, inverse class frequency weights are introduced. ( For the first (Number of samples in each class), the loss function is defined as:

[0130]

[0131] in For real labels (one-hot encoding). This loss is used to predict the probability distribution. By assigning higher weights to minority class samples, it mitigates the prediction bias caused by class imbalance and improves the diagnostic robustness of the model.

[0132] Please refer to Figure 2 The auxiliary diagnostic method based on multimodal decoupling dynamic graph learning includes the following workflow:

[0133] Data Preprocessing and Input Preparation: First, a systematic collection of heterogeneous medical data, including neuroimaging, genetic markers, and clinical assessments, was conducted to construct a multimodal medical dataset. The collected data underwent standardization to eliminate dimensional differences between different modalities. Missing values ​​were filled using appropriate imputation strategies, and outliers and noisy data were removed to ensure that the quality of all modal data met the model input standards. After data cleaning, the dataset was divided into training, validation, and test sets according to a predetermined ratio. These sets were used for model parameter training, hyperparameter tuning, and final performance evaluation, respectively, providing a reliable data foundation for subsequent full-process modeling.

[0134] Multimodal decoupling and embedding optimization: Preprocessed modal data are input one by one into the multimodal decoupling module. This module first maps the features of each modality to a common space of the same dimension using a linear projection layer, forming an intermediate feature vector. Then, a shared encoder shared by all modalities is used to extract common information across modalities, generating shared embeddings. Simultaneously, a multilayer perceptron (MLP) type specific encoder designed specifically for each modality retains the unique feature information of each modality, generating specific embeddings. During embedding generation, supervised optimization is performed using a total decoupling loss function composed of cross-orthogonal loss and cross-modal decoupling loss. This reduces information redundancy between shared and specific embeddings, standardizes the distribution of cross-modal information, and improves the purity and complementarity of the two types of embeddings, laying a solid foundation for subsequent fusion stages.

[0135] Cross-modal fusion and node representation initialization: For each subject, shared embeddings across all modalities are integrated to form a complete shared embedding sequence; simultaneously, specific embeddings corresponding to all modalities are integrated to form a specific embedding sequence. These two types of sequences are input into the multimodal fusion module, which employs a Transformer-based multi-head self-attention mechanism and introduces a specific mask matrix to avoid modal self-attention, forcing the model to focus on mining complementary information across modalities. After obtaining the enhanced shared and specific embedding sequences through attention calculation, the two types of sequences are deeply fused using a multilayer perceptron (MLP) to generate node embeddings that comprehensively represent the subject's features, serving as the initial representation of the corresponding vertices in the subject graph.

[0136] Dynamic graph modeling and iterative optimization of representation: The initial node representations of all subjects are input into a dynamic message passing module. This module first projects the node embeddings into three types of tensors—query, key, and value—using a specific attention mechanism. After normalization, a dynamic attention output containing global residuals and stabilizing factors is calculated to capture the dynamic relationships between subjects. Simultaneously, cosine similarity between subjects is calculated based on the original input features. After eliminating negative values ​​through an activation function, the most similar nodes for each subject are retained, and an initial adjacency matrix is ​​obtained after symmetric normalization. Combining the dynamic attention output and the adjacency matrix, the node embeddings are iteratively updated using hierarchical message passing. Each layer balances newly generated features with historical features through layer normalization and residual connections. After multiple rounds of iterative propagation, a fully optimized final node representation is obtained.

[0137] Diagnostic Prediction and End-to-End Model Training: A diagnostic predictor is constructed using the final optimized node representations. A class-weighted cross-entropy loss function is employed to calculate the error between the predicted results and the true labels. Class weight adjustments mitigate the potential impact of data imbalance. Based on this loss function, backpropagation is used to synchronously update all parameters of the multimodal decoupling module, multimodal fusion module, dynamic message passing module, and the predictor, achieving end-to-end joint training of the entire framework. During training, the model's diagnostic performance is monitored in real-time using a validation set, and hyperparameters such as embedding dimension, number of attention heads, and residual balance coefficient are dynamically adjusted. After model training converges, core metrics such as diagnostic accuracy and generalization ability are comprehensively evaluated on the test set, ultimately outputting stable and reliable diagnostic prediction results for brain diseases.

[0138] Compared with existing best multimodal graph learning techniques (such as MMGL, HMFGL, HierSSL, etc.), this technical solution has many significant advantages and outstanding effects, as follows:

[0139] Enhancing the efficiency of low-quality modality utilization: Through an innovative multimodal decoupling mechanism and a dedicated loss function, a clear separation between shared embedding and modality-specific embedding is achieved, which can effectively reduce noise interference in low-quality modalities, fully extract potential shared information, significantly improve the robustness of the fusion process, and successfully solve the core problem of insufficient utilization of low-quality modalities in existing methods;

[0140] Dynamic modeling of complex relationships: The innovative approach adopts a dynamic message passing strategy, which adaptively updates the similarity weights between subjects through dot product attention mechanism in each graph convolutional layer, completely breaking the limitations of static graph structure. At the same time, the combination of multi-head structure can effectively capture intra-class compactness and inter-class pathological associations, which is more in line with the dynamic characteristics of subject relationships during disease progression, and the modeling effect is more in line with clinical reality.

[0141] Improving diagnostic accuracy: In experiments related to Alzheimer's disease (TADPOLE dataset) and autism spectrum disorder (ABIDE, ABIDE-5 dataset), this technical solution showed superior performance in multi-class and binary classification tasks compared to the existing best methods. In particular, the performance gain was more significant in the new modality (such as structural MRI in ABIDE-5) scenario, which fully verified its effectiveness in actual diagnosis.

[0142] Strong generalization and robustness: In scenarios with limited labeled data (training data ratio of 10%-90%), compared with existing methods such as HMFGL and HierSSL, this technical solution demonstrates stronger robustness and generalization ability. Even with a small amount of supervised information, it can still maintain a high diagnostic accuracy, which is perfectly suited to the actual application scenarios in clinical practice where labeled data is scarce.

[0143] Revealing clinically meaningful patterns: It can effectively uncover similar patterns across different disease stages, such as successfully capturing the potential association between Alzheimer's disease (AD) and mild cognitive impairment (sMCI), and discovering key shared features in cognitive tests and MRI modalities through SHAP analysis. These findings are highly consistent with the pathological progression of AD, providing valuable insights for the study of pathological mechanisms and the stratification of disease subtypes.

[0144] Simple to implement and highly versatile: The entire framework supports end-to-end training without the need for complex external building mechanisms or integration strategies. It is easy to operate and implement, and is compatible with a variety of multimodal medical data. It is not only suitable for brain disease diagnosis scenarios, but can also be extended to related diagnostic applications in fields such as wireless and wired video image transmission. It has a wide range of applications and good prospects for promotion.

[0145] In some embodiments, the system may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the multimodal decoupling dynamic graph learning-based assisted diagnosis system may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality of auxiliary diagnosis based on multimodal decoupling dynamic graph learning.

[0146] In this embodiment, the auxiliary diagnostic system based on multimodal decoupling dynamic graph learning can be divided into multiple functional modules according to the functions it performs, such as... Figure 3 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0147] The data acquisition module is used to acquire and preprocess multimodal data from multiple subjects; the multimodal data includes at least two of the following: neuroimaging, genetic markers, and clinical assessment data.

[0148] The modality decoupling module is used to extract shared embeddings from each modality's data for each subject using a shared encoder shared by all modalities, and to extract modality-specific embeddings using a modality-specific encoder corresponding to each modality. The extraction process is optimized by a decoupling loss function to separate common pathological information across modalities from feature information unique to each modality.

[0149] The modality fusion module is used to generate an initial node representation for each subject by fusing the shared embeddings and modality-specific embeddings of all its modalities using a multi-head self-attention mechanism that introduces a mask matrix; the mask matrix is ​​used to suppress modality attention to itself.

[0150] The dynamic transmission module is used to perform hierarchical dynamic graph convolution to iteratively update the node representations based on the initial node representations of all subjects, so as to obtain optimized node representations. In each layer, the graph adjacency matrix is ​​dynamically calculated and updated based on the current node representation and the original multimodal features, and message passing and feature aggregation between nodes are performed based on the updated adjacency matrix.

[0151] The diagnostic prediction module is used to input the optimized node representations of each subject into the classifier to obtain the prediction results of the disease category.

[0152] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0153] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0154] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0156] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. An auxiliary diagnostic method based on multimodal decoupling dynamic graph learning, characterized in that, include: Multimodal data from multiple subjects were acquired and preprocessed; the multimodal data included at least two of the following: neuroimaging, genetic markers, and clinical assessment data. For each subject, shared embeddings are extracted from each modality's data using a shared encoder shared across all modalities, and modality-specific embeddings are extracted using a modality-specific encoder corresponding to each modality. The extraction process is optimized using a decoupling loss function to separate common pathological information across modalities from unique feature information of each modality. For each subject, a multi-head self-attention mechanism with an introduced mask matrix is ​​used to fuse the shared embeddings and modality-specific embeddings of all their modalities to generate the subject's initial node representation; the mask matrix is ​​used to suppress modality attention to itself; Based on the initial node representations of all subjects, hierarchical dynamic graph convolution is performed to iteratively update the node representations, resulting in optimized node representations. In each layer, the graph adjacency matrix is ​​dynamically calculated and updated based on the current node representations and the original multimodal features, and message passing and feature aggregation between nodes are performed based on the updated adjacency matrix. The optimized node representations of each subject are input into the classifier to obtain the prediction results of the disease category; Based on the initial node representations of all subjects, hierarchical dynamic graph convolution is performed to iteratively update the node representations, resulting in optimized node representations, including: For the l-th layer, the following iterative update process is performed, where the node representation of the 0-th layer is the initial node representation: The node representations of layer l are used as input and projected into query vector, key vector and value vector respectively. After normalizing the query vector and key vector, the dynamic attention output of this layer is obtained through an attention calculation function that integrates global node information as residuals. Based on the original multimodal features of all subjects, the feature similarity between each pair of subjects is calculated; the calculated similarity matrix is ​​subjected to non-negative activation and sparsification processing to retain the connection between each subject and the most similar preset number of other subjects; the processed matrix is ​​normalized to generate the dynamic adjacency matrix of the l-th layer. The intermediate update representation is obtained by fusing the dynamic attention output with the neighbor information of each node aggregated through the dynamic adjacency matrix; the intermediate update representation is then weighted and fused with the node representation of the l-th layer through residual connection, and the fusion result is subjected to layer normalization processing to finally output the node representation of the (l+1)-th layer. The iteration continues until the preset total number of graph convolutional layers L is reached. The node representation of the Lth layer output in the last iteration is used as the optimized node representation.

2. The method according to claim 1, characterized in that, Acquire multimodal data from multiple subjects and perform preprocessing, including: Multimodal raw data from multiple subjects are obtained from a pre-set medical database; the multimodal raw data includes at least neuroimaging data, genetic marker data, and clinical assessment data. The multimodal raw data is preprocessed, including: standardizing the multimodal raw data to eliminate dimensional differences between different modal data; filling in modal data with missing values; and removing outliers and noisy data.

3. The method according to claim 1, characterized in that, Shared embeddings are extracted from the modal data using a shared encoder shared by all modalities, including: For the original feature matrix of the m-th modality, a linear projection layer is used to map the original feature matrix to a common feature space of dimension D to obtain standardized intermediate features; The standardized intermediate features of each modality are input into a shared encoder shared by all modalities. The shared encoder is a multilayer perceptron (MLP). Through its parameter sharing mechanism, cross-modal common features are extracted, and the output is a shared embedding for each modality.

4. The method according to claim 1, characterized in that, Modality-specific embeddings are extracted using modality-specific encoders corresponding to each modality, including: For the standardized intermediate features of the m-th mode, input them into the mode-specific encoder dedicated to the m-th mode; The modality-specific encoder is a multilayer perceptron (MLP) independently configured for the characteristics of the modality data. It is used to extract and output the unique feature information of the modality to obtain the modality-specific embedding of the m-th modality.

5. The method according to claim 1, characterized in that, The method further includes a decoupling loss function, which comprises a cross-orthogonal loss term, a cross-modal decoupling loss term, and a total decoupling loss term obtained by a weighted sum of the two; wherein: The cross-orthogonal loss term reduces the linear correlation between shared embeddings and modality-specific embeddings of the same modality by calculating and minimizing the norm of their cross-correlation matrix. The cross-modal decoupling loss term is calculated for shared embeddings and modality-specific embeddings respectively. It maintains the information of each modality by penalizing the deviation of the diagonal elements and the unit value in the cross-modal correlation matrix, and at the same time penalizes the off-diagonal elements to suppress redundant associations of the same type of embeddings between different modalities. The total decoupling loss term is used to collaboratively optimize the cross-orthogonal loss term and the cross-modal decoupling loss term during model training, so as to drive the separation of shared embedding and modality-specific embedding.

6. The method according to claim 1, characterized in that, For each subject, a multi-head self-attention mechanism incorporating a mask matrix is ​​used to fuse the shared embeddings and modality-specific embeddings from all modalities to generate the subject's initial node representation, including: For the i-th subject, the shared embeddings of all their modalities constitute a shared embedding sequence. And the modality-specific embeddings of all its modalities constitute a specific embedding sequence. Where M is the total number of modes and D is the dimension; The shared embedding sequence and the specific embedding sequence are input into a multi-head self-attention module, which generates a query matrix Q, a key matrix K, and a value matrix V through linear projection, respectively. In the attention weight calculation, a mask matrix of size M×M is introduced, with the diagonal elements of the matrix set to negative infinity, to suppress the attention of each modality to itself; general formula Calculate the attention output, where The dimension of the attention head; The sequence output by the attention module is input into a multilayer perceptron (MLP) for nonlinear fusion and feature compression, and the initial node representation of the subject is output.

7. The method according to claim 6, characterized in that, The sequence output by the attention module is input into a multilayer perceptron (MLP) for nonlinear fusion and feature compression, outputting the subject's initial node representation, including: The optimized shared embedding sequence output by the attention module and the specific embedding sequence are concatenated along the feature dimension to obtain the concatenated feature. The spliced ​​features are input into the multilayer perceptron (MLP); the MLP performs nonlinear transformation and dimensionality reduction on the spliced ​​features and outputs the initial node representation of the subject.

8. The method according to claim 1, characterized in that, The optimized node representations of each subject are input into the classifier to obtain the disease category prediction results, including: The optimized node representation of each subject is input into a classification network. The classification network maps the node representation to the same dimension as the number of disease categories through a fully connected layer, and then processes it through a Softmax function to output the predicted probability of the subject belonging to each disease category. During the model training phase, the predicted probability is optimized using a class-weighted cross-entropy loss function, where the weight of each class is inversely proportional to the number of samples of that class in the training data, in order to alleviate the class imbalance problem.

9. An auxiliary diagnostic system based on multimodal decoupling dynamic graph learning, characterized in that, include: The data acquisition module is used to acquire and preprocess multimodal data from multiple subjects; the multimodal data includes at least two of the following: neuroimaging, genetic markers, and clinical assessment data. The modality decoupling module is used to extract shared embeddings from each modality's data for each subject using a shared encoder shared by all modalities, and to extract modality-specific embeddings using a modality-specific encoder corresponding to each modality. The extraction process is optimized by a decoupling loss function to separate common pathological information across modalities from feature information unique to each modality. The modality fusion module is used to generate an initial node representation for each subject by fusing the shared embeddings and modality-specific embeddings of all its modalities using a multi-head self-attention mechanism that introduces a mask matrix; the mask matrix is ​​used to suppress modality attention to itself. The dynamic transmission module is used to perform hierarchical dynamic graph convolution to iteratively update the node representations based on the initial node representations of all subjects, so as to obtain optimized node representations. In each layer, the graph adjacency matrix is ​​dynamically calculated and updated based on the current node representation and the original multimodal features, and message passing and feature aggregation between nodes are performed based on the updated adjacency matrix. The diagnostic prediction module is used to input the optimized node representations of each subject into the classifier to obtain the prediction results of the disease category; Based on the initial node representations of all subjects, hierarchical dynamic graph convolution is performed to iteratively update the node representations, resulting in optimized node representations, including: For the l-th layer, the following iterative update process is performed, where the node representation of the 0-th layer is the initial node representation: The node representations of layer l are used as input and projected into query vector, key vector and value vector respectively. After normalizing the query vector and key vector, the dynamic attention output of this layer is obtained through an attention calculation function that integrates global node information as residuals. Based on the original multimodal features of all subjects, the feature similarity between each pair of subjects is calculated; the calculated similarity matrix is ​​subjected to non-negative activation and sparsification processing to retain the connection between each subject and the most similar preset number of other subjects; the processed matrix is ​​normalized to generate the dynamic adjacency matrix of the l-th layer. The intermediate update representation is obtained by fusing the dynamic attention output with the neighbor information of each node aggregated through the dynamic adjacency matrix; the intermediate update representation is then weighted and fused with the node representation of the l-th layer through residual connection, and the fusion result is subjected to layer normalization processing to finally output the node representation of the (l+1)-th layer. The iteration continues until the preset total number of graph convolutional layers L is reached. The node representation of the Lth layer output in the last iteration is used as the optimized node representation.

Images