Brain age prediction method based on multimodal fusion of structural and functional MRI images

Abstract

The present invention discloses a brain age prediction method based on multimodal fusion of structural and functional MRI images, belonging to the field of medical image processing technology. The method first uses DenseNet121 to extract spatial structural features from structural magnetic resonance images; at the same time, a functional connectivity matrix is ​​constructed according to the time series of functions, and a graph structure is constructed based on the matrix, in which each node is characterized by the strength of its connection with other nodes, and the edge is converted from the absolute value of the connection strength to a sparse graph representation; then a graph attention network is used to extract functional features, and a cross-attention mechanism is used to fuse the structural and functional features; the gating mechanism fusion result is then used for the brain age prediction regression task. The brain age prediction method of the present invention makes full use of the complementary information of multimodal data and can accurately capture the biological markers of brain aging.

Description

Technical Field

[0001] The present invention relates to medical image processing technology, and in particular to a brain age prediction method based on multimodal fusion of structural and functional MRI images. Background Art

[0002] Brain health assessment is crucial for the early detection and intervention of neurodegenerative diseases. Brain age prediction, a method for measuring the difference between biological and chronological brain age, has garnered widespread attention in recent years. Magnetic resonance imaging (MRI) provides a rich source of imaging data for brain age prediction, particularly structural MRI (sMRI) and functional MRI (fMRI), which reflect brain health from the perspectives of anatomical structure and functional connectivity, respectively. Existing techniques often use deep learning methods such as convolutional neural networks (CNNs) to extract features from sMRI and predict brain age using regression models. However, relying solely on a single modality often fails to fully capture the complex physiological changes of the brain. Furthermore, fMRI provides information on dynamic functional connectivity between brain regions, but its processing is complex, often relying on traditional graph theory methods or simple fully connected feature representations, which cannot effectively capture higher-order relationships between brain regions, such as nonlinear and non-Euclidean structures.

[0003] In recent years, multimodal fusion has become an important area of ​​research in brain imaging. Although previous studies have attempted to combine structural and functional imaging, most methods have adopted simple fusion strategies and have failed to fully exploit the complementary information between modalities. This is particularly challenging when dealing with heterogeneous features between structural images and functional connectivity matrices. Summary of the Invention

[0004] Purpose of the invention: In response to the above problems, the purpose of the present invention is to provide a brain age prediction method based on multimodal fusion of structural and functional MRI images, combining structural MRI and functional MRI data to more comprehensively characterize the structural integrity and functional activity patterns of the brain, thereby improving the accuracy and robustness of brain age prediction.

[0005] Technical solution: The brain age prediction method based on multimodal fusion of structural and functional MRI images of the present invention comprises the following steps:

[0006] Obtaining original structural magnetic resonance imaging data, resampling the data, and inputting the resampled structural magnetic resonance imaging data into a convolutional neural network to obtain a structural feature map;

[0007] Flatten the structural feature graph, and obtain the structural feature label sequence after dimensionality reduction through linear mapping, and perform global averaging operation on the structural feature label sequence to obtain the structural global feature vector;

[0008] Obtain raw functional magnetic resonance imaging time series data and calculate the Pearson correlation coefficient of brain region time series;

[0009] A graph structure is constructed with brain regions as nodes and the correlations between brain regions as edges. The graph structure is input into the graph attention network to obtain the global functional features of the graph, and then the final functional features are obtained through linear mapping.

[0010] The cross-attention mechanism is used to fuse the structural feature tag sequence and the functional feature to obtain the cross-attention fusion feature;

[0011] A gated fusion mechanism is constructed to concatenate the cross-attention fusion features with the structural global feature vector, and the gating weights are calculated to obtain the fusion features, which are then input into the fully connected layer to achieve brain age prediction.

[0012] Furthermore, the step of inputting the resampled structural magnetic resonance imaging data into a convolutional neural network to obtain a structural feature map includes:

[0013] The resampled structural magnetic resonance imaging data is input into the backbone network of the DenseNet121 network, and the first set of feature maps is obtained after preprocessing through the initial convolution module; the first set of feature maps is input into the first dense block to obtain feature map f1; the feature map f1 is input into the first transition layer to obtain feature map f2; the feature map f2 is input into the second dense block to obtain feature map f3; the feature map f3 is input into the second transition layer to obtain feature map f4; the feature map f4 is input into the third dense block to obtain feature map f5; the feature map f5 is input into the third transition layer to obtain feature map f6; the feature map f6 is input into the fourth dense block to generate the structural feature map .

[0014] Furthermore, the steps of flattening the structural feature graph, obtaining a dimensionally reduced structural feature label sequence through linear mapping, and performing a global average operation on the structural feature label sequence to obtain a structural global feature vector include:

[0015] The structural feature map The channel dimension remains unchanged and is flattened along the spatial dimension into , and then converted into ,in , B represents the batch size, C represents the number of feature channels, and N represents the total number of spatial locations. Represent the depth, height and width of the feature map respectively. Each spatial position corresponds to a C-dimensional feature marker, and each feature marker is a deep spatial feature representation of a local area in the volume image.

[0016] Each feature marker is mapped to a 64-dimensional unified dimension representation through dimensional transformation, and the final shape is , which constitutes a structural feature marker sequence ;

[0017] The obtained structural feature tag sequence Perform global averaging along the spatial dimension to obtain a 64-dimensional global feature vector .

[0018] Furthermore, the steps of constructing the graph structure include:

[0019] The correlation between each brain region is used as the feature vector of the node, which represents the connection strength between the current node and other nodes. The feature vector of each node is recorded as ,in represents the connection strength between node i and node n;

[0020] A sparse graph is constructed based on the absolute connection strength. When there is no connection between nodes, the edge connection relationship is 0, otherwise the edge connection relationship is the absolute value of the connection strength. .

[0021] Furthermore, the graph structure is input into the graph attention network to obtain the global functional features of the graph, and then the steps of obtaining the final functional features through linear mapping include:

[0022] The node feature vector and the sparse graph are input into the graph attention network. The node feature vector is weighted aggregated through the first layer of graph attention convolution. Multiple attention heads are used to perform nonlinear transformation through the activation function ELU to obtain the node feature d1.

[0023] Then, the node feature d1 is extracted through the second layer of graph attention convolution, multiple attention heads are used, and nonlinear transformation is performed through the activation function ELU to obtain the node feature d2;

[0024] The third layer of graph attention convolution is used as the output layer, the node feature d2 is input to the third layer of graph attention convolution, and the single-head attention mechanism is used to compress and integrate the node features to obtain the node feature d3;

[0025] The node feature d3 is calculated through a layer of linear mapping to calculate the attention weight, and the attention weight is multiplied by the node feature element by element to obtain the weighted node feature d4;

[0026] Through the graph-level global pooling operation, the weighted node features d4 are averaged across the graph dimensions to generate a global feature representation of the entire graph;

[0027] The global feature representation is input into the linear regression layer for mapping, and the low-dimensional representation of the graph structure mode is output. , as a functional feature.

[0028] Furthermore, the steps of adopting the cross attention mechanism to fuse the structural feature tag sequence and the functional feature to obtain the cross attention fusion feature include:

[0029] Represent the structural feature tag sequence as a matrix , the matrix T and the functional features Mapping is performed, expressed as:

[0030] ,

[0031] ,

[0032] ,

[0033] Where, Represent the query vector, key matrix and value matrix in the attention mechanism respectively; Represent the learnable linear mapping weight matrices respectively; Represent the corresponding bias terms respectively;

[0034] Then the multi-head attention calculation is performed, the formula is:

[0035] ,

[0036] Where, Represents the dimension of each attention head;

[0037] The output results of multiple heads are spliced ​​together and expressed as:

[0038] ,

[0039] Where, represents the output of the mth attention head, Expressed as the output mapping matrix, Represents a splicing operation;

[0040] After the characteristics Through feedforward network processing, the final fused cross-attention fusion feature is obtained .

[0041] Furthermore, a gated fusion mechanism is constructed to concatenate the cross-attention fusion features with the structural global feature vector and calculate the gated weights. The steps to obtain the fusion features include:

[0042] Cross-attention fusion features and the structural global eigenvector Concatenate into vectors , then calculate the gating weight , the formula is:

[0043] ,

[0044] Where, represents the linear mapping parameter of the gating weight, represents the bias of the gating weight, is the Sigmoid function;

[0045] According to the gate weight calculation, the fusion feature is obtained , expressed as:

[0046] .

[0047] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0048] 1. This invention integrates structural and functional bimodal information to enhance feature expression capabilities: By fully combining structural MRI and functional MRI data, it can effectively explore the complementary characteristics between them. Compared with single-modality prediction methods, it can more comprehensively characterize the structural integrity and functional activity patterns of the brain, thereby improving the accuracy and robustness of brain age prediction;

[0049] 2. The present invention uses an improved architecture based on a two-layer multi-head graph attention network combined with dynamic attention pooling to deeply model functional connectivity features and improve the accuracy of brain age prediction: using the functional connectivity map constructed by fMRI, a three-layer progressive attention mechanism is used to deeply model the connection relationship between brain regions. This architecture achieves this through: 1) The first two layers of the multi-head graph attention network (GAT) gradually capture the brain functional network features of different scales and effectively extract the high-order nonlinear interaction patterns between local brain regions; 2) The terminal single-head compression layer focuses on the core topological features related to age; at the same time, an innovative learnable attention pooling mechanism is introduced to dynamically weight the importance of different brain regions, adaptively highlight the key brain region features that are highly correlated with brain age changes, suppress irrelevant noise interference, and further improve the robustness and accuracy of brain age prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flowchart of the brain age prediction method based on multimodal fusion of structural and functional MRI images;

[0051] Figure 2 A flowchart for extracting structural features;

[0052] Figure 3 A flowchart for extracting functional features;

[0053] Figure 4 This is the result of brain age prediction using the DenseNet121 network;

[0054] Figure 5This is the result of using graph attention network to predict brain age;

[0055] Figure 6 This is a graph showing the results of brain age prediction using the method of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings and embodiments.

[0057] Combine Figure 1 The brain age prediction method based on multimodal fusion of structural and functional MRI images described in this embodiment includes the following steps:

[0058] Step 1: Obtain original structural magnetic resonance imaging data, resample it, and input the resampled structural magnetic resonance imaging data into a convolutional neural network to obtain a structural feature map;

[0059] Furthermore, the step of inputting the resampled structural magnetic resonance imaging data into a convolutional neural network to obtain a structural feature map includes:

[0060] The resampled structural magnetic resonance imaging data is input into the backbone network of the DenseNet121 network, and the first set of feature maps is obtained after preprocessing through the initial convolution module; the first set of feature maps is input into the first dense block to obtain feature map f1; the feature map f1 is input into the first transition layer to obtain feature map f2; the feature map f2 is input into the second dense block to obtain feature map f3; the feature map f3 is input into the second transition layer to obtain feature map f4; the feature map f4 is input into the third dense block to obtain feature map f5; the feature map f5 is input into the third transition layer to obtain feature map f6; the feature map f6 is input into the fourth dense block to generate the structural feature map .

[0061] For example, structural magnetic resonance imaging data of a subject is obtained with a shape of (256, 256, 192), and is resampled to a size of (128, 128, 128), which is recorded as a T1 image.

[0062] The DenseNet121 model is selected as the convolutional neural network, the T1 image is input into the DenseNet121 model, all layers except the last layer of DenseNet121 are retained, and the final structural feature map is generated as follows: .

[0063] Combine Figure 2In the DenseNet121 model, the T1 image is preprocessed by the initial convolution module, which includes a 7×7×7 three-dimensional convolution layer, batch normalization, activation function ReLU, and 3D maximum pooling operation to extract the initial spatial features and form the first set of feature maps with a size of (32, 32, 32). The first set of feature maps is input to the first dense block, which consists of 6 bottleneck layer units, namely Figure 2 In the densely connected layer, each bottleneck layer unit performs the following operations in sequence: batch normalize the input features, introduce nonlinear transformation through the LeakyReLU activation function, use 1×1×1 convolution to compress the feature dimension, generate an intermediate feature map, apply the LeakyReLU activation function and 3×3×3 convolution operation to the intermediate feature map to extract spatial features; all bottleneck layer units are connected through feature splicing as input for subsequent processing.

[0064] The feature map f1 output by the first dense block is passed to the first transition layer, and the first transition layer executes in the following order: batch normalization is performed on the input feature map f1, LeakyReLU activation is performed on the normalized feature map, and then a 1×1×1 convolution kernel is used to compress the channel dimension. The convolution result is subjected to a 2×2×2 maximum pooling operation to achieve spatial size downsampling, and a feature map f2 of size (16, 16, 16) is obtained.

[0065] The feature map f2 of size (16, 16, 16) is input into the second dense block, which consists of 12 bottleneck layer units to further extract higher-level semantic features and obtain the feature map f3.

[0066] The output feature map f3 of the second dense block is input into the second transition layer, which has the same structure as the first transition layer. Feature compression and downsampling are completed to obtain a feature map f4 of size (8, 8, 8).

[0067] The feature map f4 of size (8, 8, 8) is input into the third dense block, and the same 24 bottleneck layer structures and splicing operations as the previous dense block are continued to extract deeper spatial features to obtain the feature map f5;

[0068] After the third dense block is completed, its output feature map f5 enters the third transition layer, and the above processing steps are repeated to achieve channel compression and spatial downsampling, obtaining a feature map f6 of size (4, 4, 4).

[0069] The feature map f6 output by the third transition layer is passed to the fourth dense block. The fourth dense block uses the same 16 bottleneck layer structure for feature extraction as the last level of structural information extraction to generate the final structural feature map .

[0070] Step 2: Flatten the structural feature graph and obtain a dimensionally reduced structural feature label sequence through linear mapping. Perform a global average operation on the structural feature label sequence to obtain a structural global feature vector.

[0071] Furthermore, the steps of flattening the structural feature graph, obtaining a dimensionally reduced structural feature label sequence through linear mapping, and performing a global average operation on the structural feature label sequence to obtain a structural global feature vector include:

[0072] The structural feature map The channel dimension remains unchanged and is flattened along the spatial dimension into , and then converted into ,in , B represents the batch size, C represents the number of feature channels, and N represents the total number of spatial locations. Represent the depth, height and width of the feature map respectively. Each spatial position corresponds to a C-dimensional feature marker, and each feature marker is a deep spatial feature representation of a local area in the volume image.

[0073] Each feature marker is mapped to a 64-dimensional unified dimension representation through dimensional transformation, and the final shape is , which constitutes a structural feature marker sequence ;

[0074] The obtained structural feature tag sequence Perform global averaging along the spatial dimension to obtain a unique 64-dimensional global feature vector .

[0075] For example, the structural feature map The channel dimension remains unchanged and is flattened along the spatial dimension into , and then converted into , each spatial position corresponds to a 1024-dimensional feature marker, and each feature marker represents the depth space feature of the local area in the volume image. All feature markers are mapped to a 64-dimensional vector of uniform dimension through dimensional transformation, and the final shape is , a sequence of local structural feature tags that constitutes the structural mode , used for subsequent cross-attention interaction and fusion with graph structure information.

[0076] The obtained structural feature marker sequence is globally averaged along the spatial dimension to obtain a unique 64-dimensional structural global feature vector , this vector is used to fuse with the multimodal fusion features processed by the cross-attention mechanism in the subsequent gated fusion module.

[0077] Step 3: Obtain the original time series data of functional magnetic resonance imaging and calculate the Pearson correlation coefficient of the brain region time series.

[0078] For example, the original time series data of functional magnetic resonance imaging of the subject is obtained and recorded as the matrix , where the number of brain regions N0=90, the number of time points T=261, and each column in the matrix X Represents the signal changes of the i-th brain region at 261 time points.

[0079] The Pearson correlation coefficient of the brain region time series was calculated using the formula:

[0080] ,

[0081] Where, represents the blood oxygen level dependent (BOLD) signal value of the i-th brain region at the t-th time point, represents the average BOLD signal value of the i-th brain region over the entire time period, represents the BOLD signal value of the jth brain region at the tth time point, represents the average BOLD signal value of the jth brain region over the entire time period;

[0082] All Forming a symmetric connection matrix , the diagonal is 1.

[0083] Step 4: Build a graph structure with brain regions as nodes and the correlations between brain regions as edges. Input the graph structure into the graph attention network to obtain the global functional features of the graph, and then obtain the final functional features through linear mapping.

[0084] Furthermore, the steps of constructing the graph structure include:

[0085] The correlation between each brain region is used as the feature vector of the node, which represents the connection strength between the current node and other nodes. The feature vector of each node is recorded as ,in represents the connection strength between node i and node n,

[0086] A sparse graph is constructed based on the absolute connection strength. When there is no connection between nodes, the edge connection relationship is 0, otherwise the edge connection relationship is the absolute value of the connection strength. .

[0087] Furthermore, the graph structure is input into the graph attention network to obtain the global functional features of the graph, and then the steps of obtaining the final functional features through linear mapping include:

[0088] The node feature vector and the sparse graph are input into the graph attention network. The node feature vector is weighted aggregated through the first layer of graph attention convolution. Multiple attention heads are used to perform nonlinear transformation through the activation function ELU to obtain the node feature d1.

[0089] Then, the node feature d1 is extracted through the second layer of graph attention convolution, multiple attention heads are used, and nonlinear transformation is performed through the activation function ELU to obtain the node feature d2;

[0090] The third layer of graph attention convolution is used as the output layer, the node feature d2 is input to the third layer of graph attention convolution, and the single-head attention mechanism is used to compress and integrate the node features to obtain the node feature d3;

[0091] The node feature d3 is calculated through a layer of linear mapping to calculate the attention weight, and the attention weight is multiplied by the node feature element by element to obtain the weighted node feature d4;

[0092] Through the graph-level global pooling operation, the weighted node features d4 are averaged across the graph dimensions to generate a global feature representation of the entire graph;

[0093] The global feature representation is input into the linear regression layer for mapping, and the low-dimensional representation of the graph structure mode is output. , as a functional feature.

[0094] For example, a node feature matrix and a sparse graph are input into a graph attention network, wherein the node feature matrix represents the initial features of each brain region node in the structural connection graph, and the edge connection relationship is determined by the structural connection strength between nodes;

[0095] Combine Figure 3 , the graph structure is recorded as graph G, including node feature vectors and sparse graphs. The first layer of graph attention (GAT) convolution performs weighted aggregation on the input node features, adopts 4 attention heads and performs nonlinear transformation through activation function ELU to enhance the feature expression ability of multi-scale neighborhood;

[0096] The second layer of graph attention convolution further extracts features from the output of the previous step, maintains the 4-head attention structure, and uses the activation function ELU to improve the structural representation capability;

[0097] The third layer of graph attention convolution is used as the output layer. It adopts a single-head attention mechanism without feature splicing to further compress and integrate node features.

[0098] An attention-based node weighting mechanism is introduced for the above node features. The operation sequence is as follows: each node feature is subjected to a linear mapping to calculate its attention weight (range is [0, 1]), and then multiplied element-by-element with the node feature to achieve importance weighting;

[0099] Through graph-level global pooling operations, all weighted node features are averaged across the graph dimensions to generate a global feature representation of the entire graph.

[0100] The global feature representation is input into the linear regression layer for mapping, and the final low-dimensional representation of the graph structure mode is output. , used for downstream multimodal fusion.

[0101] In step 5, the cross-attention mechanism is used to fuse the structural feature tag sequence and the functional feature to obtain the cross-attention fusion feature.

[0102] Furthermore, the steps of adopting the cross attention mechanism to fuse the structural feature tag sequence and the functional feature to obtain the cross attention fusion feature include:

[0103] Represent the structural feature tag sequence as a matrix , the matrix T and the functional features Mapping is performed, expressed as:

[0104] ,

[0105] ,

[0106] ,

[0107] Where, Represent the query vector, key matrix and value matrix in the attention mechanism respectively; Represent the learnable linear mapping weight matrices respectively; Represent the corresponding bias terms respectively;

[0108] Then the multi-head attention calculation is performed, the formula is:

[0109] ,

[0110] Where, Represents the dimension of each attention head;

[0111] The output results of multiple heads are spliced ​​together and expressed as:

[0112] ,

[0113] After that, the final fused cross-attention fusion feature is obtained through feedforward network processing. ,in represents the output of the mth attention head, Represented as an output mapping matrix, used to integrate the results of multiple attention heads, Represents a splicing operation.

[0114] Step 6: Construct a gated fusion mechanism to concatenate the cross-attention fusion features with the structural global feature vector, calculate the gating weights, obtain the fusion features, and input the fusion features into the fully connected layer to achieve brain age prediction.

[0115] Furthermore, a gated fusion mechanism is constructed to concatenate the cross-attention fusion features with the structural global feature vector and calculate the gated weights. The steps to obtain the fusion features include:

[0116] Cross-attention fusion features and the structural global eigenvector Concatenate into vectors , then calculate the gating weight , the formula is:

[0117] ,

[0118] Where, represents the linear mapping parameter of the gating weight, represents the bias of the gating weight, is the Sigmoid function;

[0119] According to the gate weight calculation, the fusion feature is obtained , expressed as:

[0120] .

[0121] The fusion features Input the fully connected layer and output the subject's predicted age to achieve brain age prediction.

[0122] This embodiment further illustrates the performance of the brain age prediction method of the present invention through the following examples. In this example, a total of 652 healthy subjects with an age range of 18 to 89 years old were included. A five-fold cross-validation strategy was used to train and validate the method of the present invention, with each fold trained for 50 iterations. During the training process, the batch size was 8 and the initial learning rate was , and introduces a learning rate scheduling strategy based on performance indicators. When the performance is not significantly improved in several consecutive iterations, the learning rate will decay according to the preset ratio until it reaches the minimum learning rate threshold. , in order to improve the convergence efficiency and stability of the training process.

[0123] The evaluation indicators used are mean absolute error (MAE) and Pearson correlation (PCC). The calculation formula of MAE is:

[0124] ,

[0125] Where N is the number of subject samples, is the predicted age of the subject, and y is the actual age of the subject. The evaluation results are shown in Table 1. Figures 4 to 6 As shown in the figure, an ablation experiment is conducted in this example to compare the prediction accuracy when using a single modality and the fusion of the two modal features proposed in this invention. The first method only uses structural features and uses the DenseNet121 network to extract features for brain age prediction, as shown in the figure. Figure 4 As shown; the second method only uses functional features and uses the graph attention network (FCGAT) to extract features for brain age prediction, as shown Figure 5 As shown; the third method using the method of the present invention uses the DenseNet121 network to extract structural features, uses the graph attention network to extract functional features, and then uses cross attention and gating to fuse the features of the two modalities, and then performs brain age prediction, as shown Figure 6 As shown in Figure 2, where T1 represents the T1-weighted image and FC represents the functional connectivity matrix. Ablation experiments further demonstrate that the accuracy of the brain age prediction method described in the present invention after fusion is better than that of single-modality prediction.

[0126] Table 1

[0127] .

Claims

1. A brain age prediction method based on multimodal fusion of structural and functional MRI images, characterized by: The following steps are involved: Obtaining original structural magnetic resonance imaging data, resampling the data, and inputting the resampled structural magnetic resonance imaging data into a convolutional neural network to obtain a structural feature map; Flatten the structural feature graph, and obtain the structural feature label sequence after dimensionality reduction through linear mapping, and perform global averaging operation on the structural feature label sequence to obtain the structural global feature vector; Obtain raw functional magnetic resonance imaging time series data and calculate the Pearson correlation coefficient of brain region time series; A graph structure is constructed with brain regions as nodes and the correlations between brain regions as edges. The graph structure is input into the graph attention network to obtain the global functional features of the graph, and then the final functional features are obtained through linear mapping. The cross-attention mechanism is used to fuse the structural feature tag sequence and the functional feature to obtain the cross-attention fusion feature; Construct a gated fusion mechanism to concatenate the cross-attention fusion features with the structural global feature vector, calculate the gating weights, obtain the fusion features, and input the fusion features into the fully connected layer to achieve brain age prediction; The steps to build the graph structure include: The correlation between each brain region is used as the feature vector of the node, which represents the connection strength between the current node and other nodes. The feature vector of each node is recorded as ,in represents the connection strength between node i and node n; A sparse graph is constructed based on the absolute connection strength. When there is no connection between nodes, the edge connection relationship is 0, otherwise the edge connection relationship is the absolute value of the connection strength. ; The steps of inputting the graph structure into the graph attention network to obtain the global functional features of the graph, and then obtaining the final functional features through linear mapping include: The node feature vector and the sparse graph are input into the graph attention network. The node feature vector is weighted aggregated through the first layer of graph attention convolution. Multiple attention heads are used to perform nonlinear transformation through the activation function ELU to obtain the node feature d1. Then, the node feature d1 is extracted through the second layer of graph attention convolution, multiple attention heads are used, and nonlinear transformation is performed through the activation function ELU to obtain the node feature d2; The third layer of graph attention convolution is used as the output layer, the node feature d2 is input to the third layer of graph attention convolution, and the single-head attention mechanism is used to compress and integrate the node features to obtain the node feature d3; The node feature d3 is calculated through a layer of linear mapping to calculate the attention weight, and the attention weight is multiplied by the node feature element by element to obtain the weighted node feature d4; Through the graph-level global pooling operation, the weighted node features d4 are averaged across the graph dimensions to generate a global feature representation of the entire graph; The global feature representation is input into the linear regression layer for mapping, and the low-dimensional representation of the graph structure mode is output. , as a functional feature.

2. The brain age prediction method based on multimodal fusion of structural and functional MRI images according to claim 1, characterized in that: The steps of inputting the resampled structural magnetic resonance imaging data into a convolutional neural network to obtain a structural feature map include: The resampled structural magnetic resonance imaging data is input into the backbone network of the DenseNet121 network, and the first set of feature maps is obtained after preprocessing through the initial convolution module; the first set of feature maps is input into the first dense block to obtain feature map f1; the feature map f1 is input into the first transition layer to obtain feature map f2; the feature map f2 is input into the second dense block to obtain feature map f3; the feature map f3 is input into the second transition layer to obtain feature map f4; the feature map f4 is input into the third dense block to obtain feature map f5; the feature map f5 is input into the third transition layer to obtain feature map f6; the feature map f6 is input into the fourth dense block to generate the structural feature map .

3. The brain age prediction method based on multimodal fusion of structural and functional MRI images according to claim 2, characterized in that: The steps of flattening the structural feature graph, obtaining a dimensionally reduced structural feature label sequence through linear mapping, and performing a global average operation on the structural feature label sequence to obtain a structural global feature vector include: The structural feature map The channel dimension remains unchanged and is flattened along the spatial dimension into , and then converted into ,in , B represents the batch size, C represents the number of feature channels, and N represents the total number of spatial locations. Represent the depth, height and width of the feature map respectively. Each spatial position corresponds to a C-dimensional feature marker, and each feature marker is a deep spatial feature representation of a local area in the volume image. Each feature marker is mapped to a 64-dimensional unified dimension representation through dimensional transformation, and the final shape is , which constitutes a structural feature marker sequence ; The obtained structural feature tag sequence Perform global averaging along the spatial dimension to obtain a 64-dimensional global feature vector .

4. The brain age prediction method based on multimodal fusion of structural and functional MRI images according to claim 3, characterized in that: The cross-attention mechanism is used to fuse the structural feature tag sequence and the functional feature. The steps to obtain the cross-attention fusion feature include: Represent the structural feature tag sequence as a matrix , the matrix T and the functional features Mapping is performed, expressed as: ， ， ， Where, Represent the query vector, key matrix and value matrix in the attention mechanism respectively; Represent the learnable linear mapping weight matrices respectively; Represent the corresponding bias terms respectively; Then the multi-head attention calculation is performed, the formula is: ， Where, Represents the dimension of each attention head; The output results of multiple heads are spliced ​​together and expressed as: ， Where, represents the output of the mth attention head, Expressed as the output mapping matrix, Represents a splicing operation; After the characteristics Through feedforward network processing, the final fused cross-attention fusion feature is obtained .

5. The brain age prediction method based on multimodal fusion of structural and functional MRI images according to claim 4, characterized in that: Construct a gated fusion mechanism, concatenate the cross-attention fusion features with the structural global feature vector, and calculate the gate weights. The steps to obtain the fusion features include: Cross-attention fusion features and the structural global eigenvector Concatenate into vectors , then calculate the gating weight , the formula is: ， Where, represents the linear mapping parameter of the gating weight, represents the bias of the gating weight, is the Sigmoid function; According to the gate weight calculation, the fusion feature is obtained , expressed as: 。

Images