A multi-modal brain network classification method based on feature decoupling and dynamic graph construction

Abstract

The application provides a multi-modal brain network classification method based on feature decoupling and dynamic graph construction, and relates to the technical field of artificial intelligence, and the method comprises the following steps: acquiring functional magnetic resonance imaging (fMRI) data and structural magnetic resonance imaging (sMRI) data of a to-be-tested person, obtaining multi-modal brain network data according to the fMRI data and the sMRI data, processing the multi-modal brain network data based on a preset multi-modal brain network classification model, and obtaining the brain network state of the to-be-tested person. The application can fully utilize the advantages of the two modalities by combining the fMRI data and the sMRI data, thereby improving the accuracy and comprehensiveness of the brain network state classification, and the dynamic graph attention module based on a graph attention network (GAT) can capture the dynamic change characteristics of the brain network, construct a dynamic graph representation, and enhance the sensitivity of the model to the dynamic connection relationship between brain regions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-modal brain network classification method based on feature decoupling and dynamic graph construction. BACKGROUND

[0002] In the field of neuroscience, multi-modal brain network analysis has attracted widespread attention due to its ability to provide a comprehensive view of brain structure and function. Functional magnetic resonance imaging (fMRI) can reflect blood flow changes during brain activity, while structural magnetic resonance imaging (sMRI) can reveal the anatomical structure of the brain. Traditional brain network analysis methods usually process data from the two modalities independently, which may result in the loss of key brain function and structure correlation information.

[0003] In addition, existing multi-modal fusion methods often simply linearly combine or stack data from different modalities, which makes it difficult to capture the dynamic characteristics of brain networks, i.e., the characteristics of the connection between brain regions changing over time and conditions. This method ignores the complexity and dynamics of brain network connections, limiting the sensitivity and accuracy of the model to changes in brain network state. SUMMARY

[0004] The present application solves one or more of the above related technical problems.

[0005] To solve the above problems, the present application provides a multi-modal brain network classification method based on feature decoupling and dynamic graph construction, a device and equipment.

[0006] In a first aspect, the present application provides a multi-modal brain network classification method based on feature decoupling and dynamic graph construction, comprising:

[0007] obtaining functional magnetic resonance imaging data and structural magnetic resonance imaging data of a person to be tested,

[0008] obtaining multi-modal brain network data from the functional magnetic resonance imaging data and the structural magnetic resonance imaging data, the multi-modal brain network data including at least one functional connection network matrix and one structural network matrix;

[0009] processing the multi-modal brain network data based on a pre-set multi-modal brain network classification model to obtain the brain network state of the person to be tested;

[0010] The multi-modal brain network classification model includes a feature decoupling module and a dynamic graph attention module based on GAT network construction.

[0011] Optionally, the multi-modal brain network classification model further comprises a cross-attention module and a fusion module, and the multi-modal brain network classification model based on the preset is used to process the multi-modal brain network data to obtain the brain network state of the to-be-tested person, comprising:

[0012] The feature decoupling module is used to process the functional connection network matrix and the structural network matrix respectively to obtain functional feature data, shared functional feature data, structural feature data and shared structural feature data;

[0013] The cross-attention module is used to process the shared functional feature data and the shared structural feature data to obtain a dynamic fusion graph adjacency matrix;

[0014] The dynamic graph attention module is used to process the dynamic fusion graph adjacency matrix, the functional feature data and the structural feature data to obtain a first dynamic graph and a second dynamic graph;

[0015] The fusion module is used to fuse the first dynamic graph and the second dynamic graph to obtain fusion feature data, and classify the fusion feature to obtain the brain network state of the to-be-tested person.

[0016] Optionally, the feature decoupling module comprises a first encoding module and a second encoding module; and the feature decoupling module is used to process the functional connection network matrix and the structural network matrix respectively to obtain functional feature data, shared functional feature data, structural feature data and shared structural feature data, comprising:

[0017] The first encoding module is used to process the functional connection network matrix to obtain the functional feature data and the shared functional feature data;

[0018] The second encoding module is used to process the structural network matrix to obtain the structural feature data and the shared structural feature data.

[0019] Optionally, the first encoding module comprises a feature extraction layer, a first linear projection space and a second linear projection space; and the first encoding module is used to process the functional connection network matrix to obtain the functional feature data and the shared functional feature data, comprising:

[0020] The feature extraction layer is used to extract features of the functional connection network matrix to obtain intermediate feature data;

[0021] The first linear projection space is used to map the intermediate feature data to obtain the functional feature data, and the second linear projection space is used to map the intermediate feature data to obtain the shared functional feature data.

[0022] Optionally, the processing of the shared function feature data and the shared structure feature data by the cross-attention module comprises:

[0023] respectively converting the shared function feature data and the shared structure feature data to obtain a first vector matrix and a second vector matrix, the first vector matrix comprising a first query vector and a first key vector, and the second vector matrix comprising a second query vector and a second key vector;

[0024] obtaining a second attention matrix according to the first query vector and the second key vector, and obtaining a first attention matrix according to the first key vector and the second query vector;

[0025] fusing the first attention matrix and the second attention matrix to obtain the dynamic fusion graph adjacency matrix.

[0026] Optionally, the dynamic graph attention module comprises a first GAT network, a second GAT network, a third GAT network and a fourth GAT network; and the processing of the dynamic fusion graph adjacency matrix, the function feature and the structure feature by the dynamic graph attention module to obtain a first dynamic graph and a second dynamic graph comprises:

[0027] processing the dynamic fusion graph adjacency matrix and the function feature by the first GAT network to obtain a first intermediate feature;

[0028] processing the dynamic fusion graph adjacency matrix by the second GAT network to obtain a temporary function feature;

[0029] processing the dynamic fusion graph adjacency matrix and the structure feature by the third GAT network to obtain a second intermediate feature;

[0030] processing the dynamic fusion graph adjacency matrix by the fourth GAT network to obtain a temporary structure feature;

[0031] fusing the first intermediate feature and the temporary function feature to obtain the first dynamic graph, and fusing the second intermediate feature and the temporary structure feature to obtain the second dynamic graph.

[0032] Optionally, the fusion module comprises a feedforward neural network; and the fusion processing of the first dynamic graph and the second dynamic graph by the fusion module to obtain fusion feature data comprises:

[0033] splicing the first dynamic graph and the second dynamic graph to obtain spliced vector feature data;

[0034] processing the splicing vector feature data through the feedforward neural network to obtain weight vector data;

[0035] obtaining fusion feature data according to the first dynamic graph, the second dynamic graph and the weight vector data.

[0036] Optionally, the first vector matrix comprises a first value vector, and the second vector matrix comprises a second value vector; and the processing the shared functional feature data and the shared structural feature data through the cross-attention module to obtain a dynamic fusion graph adjacency matrix further comprises:

[0037] obtaining first temporary feature data according to the first attention matrix and the first value vector;

[0038] obtaining cross-modal feature data according to the first temporary feature data and the shared functional feature data;

[0039] The cross-modal feature data is used to enhance an auxiliary output of the multi-modal brain network classification model training.

[0040] In a second aspect, the present application provides a multi-modal brain network classification device based on feature decoupling and dynamic graph construction, comprising:

[0041] An acquisition unit is configured to acquire functional magnetic resonance imaging (fMRI) data and structural magnetic resonance imaging (sMRI) data of a person to be tested.

[0042] A construction unit is configured to obtain multi-modal brain network data according to the fMRI data and the sMRI data, wherein the multi-modal brain network data comprises at least one functional connection network matrix and one structural network matrix.

[0043] A processing unit is configured to process the multi-modal brain network data based on a preset multi-modal brain network classification model to obtain a brain network state of the person to be tested, wherein the multi-modal brain network classification model comprises a feature decoupling module and a dynamic graph attention module based on a GAT network.

[0044] In a third aspect, the present application provides a multi-modal brain network classification device based on feature decoupling and dynamic graph construction, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-modal brain network classification method based on feature decoupling and dynamic graph construction as described in the first aspect.

[0045] The multi-modal brain network classification method, device and equipment based on feature decoupling and dynamic graph construction have the following advantages:

[0046] By combining functional magnetic resonance imaging (fMRI) and structural magnetic resonance imaging (sMRI) data, the advantages of both modalities can be fully utilized, thereby improving the accuracy and comprehensiveness of brain network state classification. The feature decoupling module can separate unique and shared features in the functional and structural network matrices, reducing information interference between modalities and thus improving feature purity and expressive power. The dynamic graph attention module based on graph attention networks (GAT) can capture the dynamic changes in brain networks, construct dynamic graph representations, and enhance the model's sensitivity to dynamic connectivity relationships between brain regions. These features enhance the model's robustness and generalization ability, enabling it to better adapt to complex brain network data. Furthermore, the attention mechanism of the dynamic graph attention module provides interpretability for the model's decision-making process, helping to understand the role and contribution of different brain regions in specific brain network states. Overall, this invention demonstrates significant advantages in brain network classification tasks and can provide more reliable technical support for the diagnosis and research of brain diseases. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating a multimodal brain network classification method based on feature decoupling and dynamic graph construction according to an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of the structure of a multimodal brain network classification model according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the feature decoupling module processing procedure in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the cross-modal attention module according to an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of the fusion module in an embodiment of the present invention. Detailed Implementation

[0052] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0053] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0054] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0055] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0056] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0057] To address the problems existing in the aforementioned related technologies, this invention provides a multimodal brain network classification method based on feature decoupling and dynamic graph construction.

[0058] like Figure 1 As shown in the figure, an embodiment of the present invention provides a multimodal brain network classification method based on feature decoupling and dynamic graph construction, comprising:

[0059] Step S100: Acquire functional magnetic resonance imaging (fMRI) data and structural magnetic resonance imaging (SMRI) data of the subject.

[0060] Specifically, fMRI and sMRI data of subjects are typically acquired using magnetic resonance imaging (MRI) equipment. fMRI primarily records oxygenation-dependent signals in the brain, reflecting the dynamics of neural activity; sMRI, on the other hand, acquires information about the brain's anatomical structure and distinguishes brain region boundaries. Combining the two allows multimodal brain network classification models to utilize both static anatomical and dynamic functional connectivity information, improving classification accuracy and robustness, and providing a powerful tool for brain disease research and clinical applications.

[0061] Step S200: Obtain multimodal brain network data based on the functional magnetic resonance imaging data and the structural magnetic resonance imaging data. The multimodal brain network data includes at least one functional connectivity network matrix and one structural network matrix.

[0062] Specifically, the fMRI data is first denoised to reduce interference from cardiac signals, head movement artifacts, etc. Then, the functional connectivity network matrix can be calculated using methods such as Pearson correlation coefficient or sliding window correlation. This matrix reflects the dynamic connectivity between brain functional regions. For sMRI data, diffusion tensor imaging (DTI) and fiber tract tracking algorithms can be used to capture the connectivity of white matter fiber tracts. A structural network matrix is ​​then constructed based on edge weight formulas in graph theory, describing the physical connectivity between brain regions.

[0063] In some embodiments, such as Figure 2 As shown, the structural network matrix can be constructed using sMRI data or DTI data respectively, while multimodal brain network data includes one functional connectivity network matrix and two structural network matrices.

[0064] Functional connectivity matrices capture the dynamic characteristics of brain region functional activity, aiding in the identification of complex neural activity patterns; structural network matrices provide high-precision information on the physical connections of brain regions, avoiding the influence of imaging artifacts. Integrating these two types of networks into multimodal brain network data not only enriches feature representations but also improves the performance of subsequent classification tasks, enabling the model to more comprehensively understand the state of brain networks.

[0065] Step S300: Based on a preset multimodal brain network classification model, the multimodal brain network data is processed to obtain the brain network state of the person under test; wherein, the multimodal brain network classification model includes a feature decoupling module and a dynamic graph attention module constructed based on a GAT network.

[0066] Specifically, the feature decoupling module first inputs the multimodal brain network data (functional connectivity network matrix and structural network matrix) into independent feature extraction networks (such as independent encoders) to extract features from each modality. Then, through linear or nonlinear projection, the decoupled features are mapped to independent and shared feature spaces, refining information and separating commonalities from specific features, while also preserving potential collaborative relationships between modalities. The dynamic graph attention module, based on the decoupled features, uses a dynamic graph generator to dynamically adjust the graph structure of the brain network according to the similarity between features, generating a dynamic graph. Then, through an attention mechanism based on GAT networks, it calculates the attention weights between nodes, weightedly aggregating brain region features to capture dynamic connectivity relationships and key connection patterns between brain regions. The brain network state includes normal and abnormal states. Furthermore, it includes multi-level (modal level, brain connectivity level) interpretability information to reveal the model's decision-making basis.

[0067] In this process, the feature decoupling module avoids intermodal interference and improves the accuracy of feature representation; the combination of dynamic graph generator and GAT network enhances the ability to capture dynamic connectivity relationships in brain regions; the overall design improves the robustness and accuracy of the model in classifying brain network states, providing a more reliable solution for complex brain network analysis.

[0068] In this embodiment, by combining functional magnetic resonance imaging (fMRI) and structural magnetic resonance imaging (sMRI) data, the advantages of both modalities can be fully utilized, thereby improving the accuracy and comprehensiveness of brain network state classification. The feature decoupling module can separate unique and common features in the functional and structural network matrices, reducing information interference between modalities and thus improving feature purity and expressive power. The dynamic graph attention module based on graph attention networks (GAT) can capture the dynamic changes in brain networks, construct dynamic graph representations, and enhance the model's sensitivity to dynamic connectivity relationships between brain regions. These features enhance the model's robustness and generalization ability, enabling it to better adapt to complex brain network data. Furthermore, the attention mechanism of the dynamic graph attention module provides interpretability for the model's decision-making process, helping to understand the role and contribution of different brain regions in specific brain network states. Overall, this invention demonstrates significant advantages in brain network classification tasks and can provide more reliable technical support for the diagnosis and research of brain diseases.

[0069] Optionally, the multimodal brain network classification model further includes a cross-attention module and a fusion module. The process of processing the multimodal brain network data based on the preset multimodal brain network classification model to obtain the brain network state of the subject includes:

[0070] The feature decoupling module processes the functional connection network matrix and the structural network matrix respectively to obtain functional feature data, shared functional feature data, structural feature data, and shared structural feature data.

[0071] The shared functional feature data and the shared structural feature data are processed by the cross-attention module to obtain a dynamic fusion graph adjacency matrix;

[0072] The dynamic graph attention module processes the adjacency matrix of the dynamic fusion graph, the functional feature data, and the structural feature data to obtain a first dynamic graph and a second dynamic graph.

[0073] The fusion module performs fusion processing on the first dynamic image and the second dynamic image to obtain fused feature data, and classifies the fused features to obtain the brain network state of the test subject.

[0074] Specifically, such as Figure 2 As shown, the model also includes a cross-attention module and a fusion module to improve the accuracy and interpretability of classification.

[0075] A feature decoupling module is used to process the functional connectivity network matrix (from fMRI data) and the structural network matrix (from sMRI data) separately. Specific features (functional and structural feature data) and shared features (shared functional and shared structural feature data) are extracted from each modality. The aim is to refine the information, separating the commonalities and unique characteristics of each modality, in order to more accurately capture and utilize key information from the multimodal data.

[0076] By processing shared feature data across attention modules, correlations between different modalities can be learned. This processing yields a dynamic fusion graph adjacency matrix that characterizes the interactions between functional and structural features.

[0077] The dynamic graph attention module uses a dynamic fusion graph adjacency matrix and specific feature data to generate two dynamic graphs: a first dynamic graph (based on functional features) and a second dynamic graph (based on structural features). These dynamic graphs can capture the dynamic characteristics of brain networks and reflect the connection patterns between brain regions in different states.

[0078] The fusion module merges the features of the first dynamic image and the second dynamic image to obtain fused feature data.

[0079] Finally, the fusion features are classified to determine the brain network status of the test subjects.

[0080] Feature decoupling and cross-attention mechanisms enhance the model's representation of multimodal data features, improving feature discriminative power. The use of dynamic graph attention and fusion modules enables the model to accurately capture brain network dynamics, enhancing classification accuracy. Cross-attention and fusion modules endow the model with the ability to explain the contributions of different modalities to classification decisions, improving interpretability. Furthermore, the model's dynamic graph construction and adaptive attention fusion mechanism allow it to adapt to different brain network states, significantly improving generalization ability.

[0081] Optionally, the feature decoupling module includes a first encoding module and a second encoding module; the step of processing the functional connectivity network matrix and the structural network matrix through the feature decoupling module to obtain functional feature data, shared functional feature data, structural feature data, and shared structural feature data includes:

[0082] The first encoding module processes the functional connection network matrix to obtain the functional feature data and the shared functional feature data.

[0083] The second encoding module processes the structural network matrix to obtain the structural feature data and the shared structural feature data.

[0084] Optionally, the first encoding module includes a feature extraction layer, a first linear projection space, and a second linear projection space; the step of processing the functional connection network matrix through the first encoding module to obtain the functional feature data and the shared functional feature data includes:

[0085] The feature extraction layer extracts features from the functional connectivity network matrix to obtain intermediate feature data.

[0086] The intermediate feature data is mapped through the first linear projection space to obtain the functional feature data, and the intermediate feature data is mapped through the second linear projection space to obtain the shared functional feature data.

[0087] Specifically, the feature decoupling module processes the functional connectivity network matrix and the structural network matrix through a first encoding module and a second encoding module, respectively. The first encoding module includes a feature extraction layer, a first linear projection space, and a second linear projection space. The feature extraction layer first extracts intermediate feature data from the functional connectivity network matrix. Then, the first linear projection space maps the intermediate features to functional feature data, and the second linear projection space maps the intermediate features to shared functional feature data. Similarly, the second encoding module processes the structural network matrix to obtain structural feature data and shared structural feature data. This process helps improve the discriminative power of feature representations, increases classification accuracy, and enhances the interpretability of the model.

[0088] In some embodiments, such as Figure 3 The diagram shows the processing procedure of the feature decoupling module.

[0089] The input data includes brain network matrices of two modalities. (Functional connection network matrix) and (Structure network matrix), representing functional and structural modes respectively.

[0090] The input data for each modality is processed by the feature extraction module. This step may include preprocessing operations such as denoising and normalization.

[0091] Residual layer processing: The extracted feature data is further processed through the residual layer to enhance the model's ability to learn features.

[0092] Feature decoupling: The functional connection network matrix is ​​decoupled through the first encoding module. The data is processed to obtain functional feature data. and shared functional feature data .

[0093] The structural network matrix is ​​processed through the second encoding module. The data is processed to obtain structural feature data. and shared structural feature data The second encoding module includes a corresponding feature extraction layer, a third linear projection space, and a fourth linear projection space.

[0094] Feature projection: through the first linear projection space ( Figure 3 (The corresponding special projection head) for intermediate functional features Mapping is performed to obtain functional feature data. and through the second linear projection space ( Figure 3 (corresponding common projection head) for intermediate functional features Mapping is performed to obtain shared functional feature data. .

[0095] Through the third linear projection space ( Figure 3 (The corresponding unique projection head) for intermediate structural features Mapping is performed to obtain structural feature data. Through the third and fourth linear projection space ( Figure 3 (corresponding common projection head) for intermediate structural features Mapping is performed to obtain shared structural feature data. .

[0096] During training, maximum mean difference (MMD) loss is introduced. To ensure that shared features across different modalities have similar probability distributions, this forces the encoder to reach a "consensus" at the output of the common projection head. Simultaneously, orthogonal loss is introduced. To ensure that shared and unique features are as uncorrelated as possible within each modality, MMD forces them to tend towards geometric orthogonality in the feature space. Specifically, MMD implicitly maps features to a high-dimensional Reproducing Kernel Hilbert Space (RKHS) using a kernel function (e.g., a Gaussian radial basis function kernel) and calculates the distance between the mean elements of two distributed elements in this space. By minimizing this distance, the system forces the two independent encoders to learn to reach a "consensus" at the outputs of their respective common projection heads (corresponding to the second and fourth linear projection spaces), i.e., outputting statistically indistinguishable features. Simultaneously, orthogonality loss is used to ensure that shared and unique features are as uncorrelated as possible within each modality. This loss is achieved by calculating and minimizing the square of the cosine similarity between the shared and unique feature vectors, thereby forcing them to tend towards geometric orthogonality in the feature space and ensuring the "purity" of decoupling.

[0097] Model training and optimization: The model is trained by minimizing the loss function through backpropagation and optimization algorithms such as Adam or SGD.

[0098] This process, through feature decoupling and cross-modal attention mechanisms, enables the model to more fully express the features of multimodal data, improves the discriminative power of features, thereby improving classification accuracy and providing modality-level interpretation, thus enhancing the interpretability of the model.

[0099] Optionally, the step of processing the shared functional feature data and the shared structural feature data through the cross-attention module to obtain the dynamic fusion graph adjacency matrix includes:

[0100] The shared functional feature data and the shared structural feature data are transformed respectively to obtain a first vector matrix and a second vector matrix. The first vector matrix includes a first query vector and a first key vector, and the second vector matrix includes a second query vector and a second key vector.

[0101] The second attention matrix is ​​obtained based on the first query vector and the second key vector, and the first attention matrix is ​​obtained based on the first key vector and the second query vector.

[0102] The first attention matrix and the second attention matrix are fused to obtain the adjacency matrix of the dynamic fusion graph.

[0103] Optionally, the first vector matrix includes a first value vector, and the second vector matrix includes a second value vector; the step of processing the shared functional feature data and the shared structural feature data through the cross-attention module to obtain the dynamic fusion graph adjacency matrix further includes:

[0104] The first temporary feature data is obtained based on the first attention matrix and the first value vector;

[0105] Cross-modal feature data is obtained based on the first temporary feature data and the shared functional feature data;

[0106] The cross-modal feature data is used to enhance the auxiliary output of the training of the multimodal brain network classification model.

[0107] In some embodiments, such as Figure 4 The diagram illustrates the cross-modal attention module. The input to this module is the shared feature matrix (including shared functional feature data and shared structural feature data) from the decoupled module. A bidirectional cross-modal attention mechanism is employed to learn the interaction patterns between nodes. This mechanism is implemented using a standard multi-head attention layer, its principle derived from the Transformer architecture. The advantage of the multi-head mechanism is that it allows the model to learn multiple different attention patterns in parallel across different representation subspaces, thereby capturing richer intermodal relationships.

[0108] Specifically, three independent linear projection matrices are first learned for each attention head and each modality's shared features, and then transformed into queries (query vectors) respectively. Key (key vector) Sum of values ​​(Value vector) These three different projections allow the model to independently learn "what to focus on (Query)," "how to be focused on (Key)," and "what information to provide (Value)."

[0109] Subsequently, bidirectional attention computation is performed: in the first direction, a query of one modality... (The first query vector corresponding to the shared functional feature data) and the key of another modality (The second key vector corresponding to the shared structural feature data) is subjected to scaled dot-product attention, which is to calculate the matrix product. This is then divided by a scaling factor (typically the square root of the key vector dimension) to prevent the gradient from becoming too small, and then passed through the Softmax function to obtain the attention score (the second attention matrix). This score is subsequently used to adjust the corresponding values. The second value vector is weighted and summed. This process is performed in parallel across all attention heads, and the results are concatenated and transformed again through a linear layer to obtain a functional feature representation (first temporary feature data or second temporary feature data) that includes structural modality context information. The exact same operation is performed in the second direction, i.e., the first attention matrix is ​​obtained by using the first key vector corresponding to the shared functional feature data and the second query vector corresponding to the shared structural feature data. The first attention matrix is ​​then compared with the corresponding values... The first value vector is weighted and summed to obtain the corresponding structural feature representation containing structural modal context information.

[0110] The attention weight matrices (first attention matrix and second attention matrix) generated from the attention calculations in these two directions are then symmetrically fused (e.g., taking the average of the weights in both directions) and subjected to final row-wise Softmax normalization to generate a dynamic fusion graph adjacency matrix tailored to the current test subject. .

[0111] Furthermore, in order to further enhance the learning of intermodal representations, such as Figure 4 As shown, the output of the attention mechanism (first temporary feature data or second temporary feature data) is combined with the original vector (corresponding shared functional feature data or shared structural functional data) through a residual connection, and then processed by subsequent layer normalization and a feedforward network (composed of linear layers) for a cross-modal feature generation task (corresponding cross-modal feature data). This task is supervised by an auxiliary generation loss (such as mean squared error loss), prompting the model to learn more meaningful and generalizable cross-modal shared representations. The second temporary feature data is obtained based on the second attention matrix and the second value vector.

[0112] The cross-attention module captures dynamic interactions between different modalities, generating an adjacency matrix containing task-aware information that truly reflects the dynamic characteristics of brain networks. The dynamic graph adjacency matrix adapts to dynamic connectivity changes between brain regions, avoiding the limitations of fixed-modal relationships. Independent query and key-value projection mechanisms enhance the model's ability to represent features from different modalities. Generating task and residual connections strengthens feature learning, improves feature robustness and generalization ability, and provides reliable feature representations for subsequent classification.

[0113] Optionally, the dynamic graph attention module includes a first GAT network, a second GAT network, a third GAT network, and a fourth GAT network; the step of processing the adjacency matrix of the dynamic fusion graph, the functional features, and the structural features through the dynamic graph attention module to obtain the first dynamic graph and the second dynamic graph includes:

[0114] The first intermediate feature is obtained by processing the adjacency matrix of the dynamic fusion graph and the functional features through the first GAT network;

[0115] Temporary functional features are obtained by processing the adjacency matrix of the dynamic fusion graph through the second GAT network;

[0116] The adjacency matrix of the dynamic fusion graph and the structural features are processed by the third GAT network to obtain the second intermediate feature;

[0117] The adjacency matrix of the dynamic fusion graph is processed by the fourth GAT network to obtain temporary structural features;

[0118] The first intermediate feature and the temporary functional feature are fused to obtain the first dynamic graph, and the second intermediate feature and the temporary structural feature are fused to obtain the second dynamic graph.

[0119] Specifically, the dynamic graph attention module uses four graph attention networks (GAT) to process the dynamically fused graph adjacency matrix as well as functional and structural features to generate two dynamic graphs.

[0120] The first GAT network: This network dynamically fuses the graph adjacency matrix and functional features to generate first intermediate features. These features contain functional feature information weighted by a graph attention mechanism.

[0121] The second GAT network: This network processes the dynamically fused graph adjacency matrix separately to generate temporary functional features. This helps to extract functional feature information based solely on the graph structure.

[0122] The third GAT network: This network dynamically fuses the graph adjacency matrix and structural features to generate second intermediate features. These features contain structural feature information weighted by the graph attention mechanism.

[0123] The fourth GAT network: This network processes the dynamically fused graph adjacency matrix separately to generate temporary structural features. This helps to extract structural feature information based solely on the graph structure.

[0124] Feature fusion: The first intermediate feature is fused with the temporary functional feature to generate a first dynamic graph. This dynamic graph integrates functional features and graph structure information. Similarly, the second intermediate feature is fused with the temporary structural feature to generate a second dynamic graph, which integrates structural features and graph structure information.

[0125] In some embodiments, such as Figure 2 As shown, the adjacency matrix of the dynamic graph generated in the previous step... As the underlying topology of the Graph Attention Network (GAT), it defines the neighborhood relationships for information transmission. Next, the functional feature data separated from the feature decoupling module... and structural feature data These serve as the initial node features for two independent GAT flows. Within each GAT network, information aggregation is iteratively performed through a multi-layered GAT architecture containing residual connections. The GAT utilizes its self-attention mechanism to aggregate neighbors (from...). When defining information, different weights are assigned to different neighbors. These weights are learned end-to-end, enabling the model to amplify signals from important neighbors and suppress noise. As a result, modality-specific features are refined and contextualized, generating high-level, context-aware dynamic graph representations for subsequent decision-making.

[0126] By processing multiple GAT networks, this model can more fully extract information from the graph structure, thereby enhancing the expressive power of features. The step of fusing intermediate and temporary features integrates information from different sources, generating richer dynamic graph representations. Generating two dynamic graphs, one corresponding to functional features and the other to structural features, helps to distinguish and understand the roles of different modalities in the brain network. This more accurately captures the dynamic characteristics of the brain network, improving the performance of subsequent classification tasks. Furthermore, since the dynamic graphs represent functional and structural features respectively, this increases the interpretability of the model's decision-making process.

[0127] Optionally, the fusion module includes a feedforward neural network; the step of fusing the first dynamic graph and the second dynamic graph through the fusion module to obtain fused feature data includes:

[0128] The first dynamic image and the second dynamic image are spliced ​​together to obtain spliced ​​vector feature data.

[0129] The feedforward neural network is used to process the concatenated vector feature data to obtain weight vector data;

[0130] The fused feature data is obtained based on the first dynamic graph, the second dynamic graph, and the weight vector data.

[0131] Specifically, such as Figure 5 The diagram shows the structure of the fusion module. The input to the fusion module is the first and second dynamic graphs obtained from the dynamic graph attention module. First, these are aggregated into a graph-level vector through graph readout operations (such as global average pooling). and The two graph-level vectors are concatenated by a concatenation layer. The concatenated high-dimensional vector is then fed into a small feedforward neural network (represented by a linear layer in the diagram). This network acts as a weight generator, learning a two-dimensional vector (weight vector data) that represents the relative importance of the two modalities in the current sample. This vector is then normalized using a softmax function, generating two scalar values ​​between 0 and 1 that sum to 1. These are the attention weights or modality contribution scores. Ultimately, these two weights are used to perform a weighted summation of the two graph-level vectors from the original input, resulting in a final, highly condensed fusion feature vector (fusion feature data). This weighting mechanism is dynamic and sample-specific. This fusion vector is then fed into the final classifier for diagnosis. Crucially, the attention weights output by this module are themselves a form of interpretability, clearly quantifying the model's dependence on different modal information when diagnosing specific samples.

[0132] This fusion module, by splicing feature data from the first and second dynamic graphs, integrates information from different modalities to generate a comprehensive feature representation. This helps capture more comprehensive brain network features and improves the model's understanding of complex brain network states. In terms of weight learning, the feedforward neural network learns and generates weight vector data, automatically determining the relative importance of each modality in the current sample. This allows the model to dynamically adjust its dependence on information from different modalities, improving its adaptability and accuracy. Regarding feature condensation, the final generated fused feature data is a highly condensed information representation, combining key features from both dynamic graphs. This reduces data dimensionality while enhancing feature expressiveness, which is beneficial for improving classifier performance. In terms of interpretability, the output attention weights provide interpretability for the model's decision-making process. By quantifying the model's dependence on information from different modalities, users can more clearly understand the model's diagnostic basis for specific samples, enhancing the model's transparency and credibility. In terms of dynamic adaptation, the fusion process is dynamic and sample-specific. The model can adjust the fusion strategy according to different input data, making it more flexible and robust when processing diverse brain network data. In summary, this fusion module significantly improves the performance and reliability of the model in brain network state classification tasks by integrating multimodal information, learning weights, condensing features, and providing interpretability.

[0133] This invention provides a multimodal brain network classification device based on feature decoupling and dynamic graph construction, comprising:

[0134] The acquisition unit is used to acquire functional magnetic resonance imaging (fMRI) data and structural magnetic resonance imaging (SMRI) data of the subject.

[0135] A construction unit is used to obtain multimodal brain network data based on the functional magnetic resonance imaging data and the structural magnetic resonance imaging data, wherein the multimodal brain network data includes at least one functional connectivity network matrix and one structural network matrix.

[0136] The processing unit is used to process the multimodal brain network data based on a preset multimodal brain network classification model to obtain the brain network state of the person under test; wherein, the multimodal brain network classification model includes a feature decoupling module and a dynamic graph attention module constructed based on a GAT network.

[0137] This invention provides a multimodal brain network classification device based on feature decoupling and dynamic graph construction, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to implement the multimodal brain network classification method based on feature decoupling and dynamic graph construction as described above when the computer program is executed.

[0138] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the multimodal brain network classification method based on feature decoupling and dynamic graph construction as described above.

[0139] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A multi-modal brain network classification method based on feature decoupling and dynamic graph construction, characterized in that, The method comprises the following steps: obtaining functional magnetic resonance imaging data and structural magnetic resonance imaging data of a to-be-tested person, obtaining multi-modal brain network data from the functional magnetic resonance imaging data and the structural magnetic resonance imaging data, the multi-modal brain network data comprising at least one functional connection network matrix and one structural network matrix; processing the multi-modal brain network data based on a preset multi-modal brain network classification model to obtain a brain network state of the to-be-tested person; wherein the multi-modal brain network classification model comprises a feature decoupling module and a dynamic graph attention module constructed based on a GAT network, and further comprises a cross-attention module and a fusion module; the processing of the multi-modal brain network data based on the preset multi-modal brain network classification model to obtain the brain network state of the to-be-tested person comprises: processing the functional connection network matrix and the structural network matrix respectively by the feature decoupling module to obtain functional feature data, shared functional feature data, structural feature data and shared structural feature data; processing the shared functional feature data and the shared structural feature data by the cross-attention module to obtain a dynamic fusion graph adjacency matrix; wherein the dynamic graph attention module comprises a first GAT network, a second GAT network, a third GAT network and a fourth GAT network; processing the dynamic fusion graph adjacency matrix, the functional feature data and the structural feature data by the dynamic graph attention module to obtain a first dynamic graph and a second dynamic graph, comprising: processing the dynamic fusion graph adjacency matrix and the functional feature data by the first GAT network to obtain a first intermediate feature; processing the dynamic fusion graph adjacency matrix by the second GAT network to obtain temporary functional feature; processing the dynamic fusion graph adjacency matrix and the structural feature data by the third GAT network to obtain a second intermediate feature; processing the dynamic fusion graph adjacency matrix by the fourth GAT network to obtain temporary structural feature; fusing the first intermediate feature and the temporary functional feature to obtain the first dynamic graph, and fusing the second intermediate feature and the temporary structural feature to obtain the second dynamic graph; fusing the first dynamic graph and the second dynamic graph by the fusion module to obtain fusion feature data, and classifying the fusion feature to obtain the brain network state of the to-be-tested person.

2. The multi-modal brain network classification method based on feature decoupling and dynamic graph construction according to claim 1, characterized in that, The feature decoupling module comprises a first encoding module and a second encoding module; the processing of the functional connection network matrix and the structural network matrix by the feature decoupling module to obtain the functional feature data, the shared functional feature data, the structural feature data and the shared structural feature data comprises: processing the functional connection network matrix by the first encoding module to obtain the functional feature data and the shared functional feature data; processing the structural network matrix by the second encoding module to obtain the structural feature data and the shared structural feature data.

3. The multi-modal brain network classification method based on feature decoupling and dynamic graph construction according to claim 2, characterized in that, The first encoding module comprises a feature extraction layer, a first linear projection space and a second linear projection space; The processing of the functional connectivity network matrix by the first encoding module comprises: feature extraction of the functional connectivity network matrix by the feature extraction layer to obtain intermediate feature data; mapping processing of the intermediate feature data by the first linear projection space to obtain the functional feature data, and mapping processing of the intermediate feature data by the second linear projection space to obtain the shared functional feature data. 4.The multi-modal brain network classification method based on feature decoupling and dynamic graph construction of claim 1, wherein, The processing of the shared functional feature data and the shared structural feature data by the cross-attention module comprises: conversion of the shared functional feature data and the shared structural feature data to obtain a first vector matrix and a second vector matrix, the first vector matrix comprising a first query vector and a first key vector, and the second vector matrix comprising a second query vector and a second key vector; obtaining a second attention matrix according to the first query vector and the second key vector, and obtaining a first attention matrix according to the first key vector and the second query vector; fusion of the first attention matrix and the second attention matrix to obtain the dynamic fusion graph adjacency matrix.

5. The multi-modal brain network classification method based on feature decoupling and dynamic graph construction according to claim 1, characterized in that, The fusion module comprises a feedforward neural network. The fusion processing of the first dynamic graph and the second dynamic graph by the fusion module comprises: splicing processing of the first dynamic graph and the second dynamic graph to obtain spliced vector feature data; processing of the spliced vector feature data by the feedforward neural network to obtain weight vector data; obtaining fusion feature data according to the first dynamic graph, the second dynamic graph and the weight vector data.

6. The multi-modal brain network classification method based on feature decoupling and dynamic graph construction according to claim 4, characterized in that, The first vector matrix comprises a first value vector, and the second vector matrix comprises a second value vector; the processing of the shared functional feature data and the shared structural feature data by the cross-attention module to obtain a dynamic fusion graph adjacency matrix further comprises: obtaining first temporary feature data according to the first attention matrix and the first value vector; obtaining cross-modal feature data according to the first temporary feature data and the shared functional feature data; The cross-modal feature data is used to enhance the auxiliary output of the multi-modal brain network classification model training.

7. A multi-modal brain network classification apparatus based on feature decoupling and dynamic graph construction, characterized in that, comprises: an acquisition unit configured to acquire functional magnetic resonance imaging data and structural magnetic resonance imaging data of a person to be tested, a construction unit configured to obtain multi-modal brain network data from the functional magnetic resonance imaging data and the structural magnetic resonance imaging data, the multi-modal brain network data comprising at least one functional connectivity network matrix and one structural network matrix; The processing unit is configured to process the multi-modal brain network data based on a preset multi-modal brain network classification model to obtain the brain network state of the to-be-tested person, wherein the multi-modal brain network classification model comprises a feature decoupling module and a dynamic graph attention module constructed based on a GAT network, and further comprises a cross-attention module and a fusion module; the processing of the multi-modal brain network data based on the preset multi-modal brain network classification model to obtain the brain network state of the to-be-tested person comprises: processing the functional connectivity network matrix and the structural network matrix respectively by the feature decoupling module to obtain functional feature data, shared functional feature data, structural feature data and shared structural feature data; processing the shared functional feature data and the shared structural feature data by the cross-attention module to obtain a dynamic fusion graph adjacency matrix; wherein the dynamic graph attention module comprises a first GAT network, a second GAT network, a third GAT network and a fourth GAT network; processing the dynamic fusion graph adjacency matrix, the functional feature data and the structural feature data by the dynamic graph attention module to obtain a first dynamic graph and a second dynamic graph, comprising: processing the dynamic fusion graph adjacency matrix and the functional feature data by the first GAT network to obtain a first intermediate feature; processing the dynamic fusion graph adjacency matrix by the second GAT network to obtain temporary functional feature; processing the dynamic fusion graph adjacency matrix and the structural feature data by the third GAT network to obtain a second intermediate feature; processing the dynamic fusion graph adjacency matrix by the fourth GAT network to obtain temporary structural feature; fusing the first intermediate feature and the temporary functional feature to obtain the first dynamic graph, and fusing the second intermediate feature and the temporary structural feature to obtain the second dynamic graph; fusing the first dynamic graph and the second dynamic graph by the fusion module to obtain fusion feature data, and classifying the fusion feature to obtain the brain network state of the to-be-tested person.

8. A multi-modal brain network classification device based on feature decoupling and dynamic graph construction, comprising a memory, a processor and a computer program stored on the memory and capable of running on the processor, characterized in that, The processor executes the computer program to implement the multi-modal brain network classification method based on feature decoupling and dynamic graph construction according to any one of claims 1 to 6.

Images