Brain Age Prediction Method, System, Device and Medium Based on Multimodal Information Fusion

Abstract

The present invention relates to the technical fields of brain age prediction and image data processing, and specifically provides a brain age prediction method, system, device and medium based on multi-modal information fusion. The preprocessed image is subjected to multi-scale enhancement processing, and by combining multi-scale features and spatial attention information, a brain image feature map is obtained. Different feature extraction methods are used for the brain region node feature matrix and the adjacency feature matrix respectively. Among them, several different graph convolutions are used for parallel convolution processing to extract brain region node features, and a combination of edge convolution processing and row-column convolution processing is used to extract adjacency features. After splicing, a radiomics feature map of the brain is obtained, and further feature splicing and cross-modal feature fusion processing are performed on the brain image feature map and the radiomics feature map of the brain to achieve accurate prediction of the brain age.

Description

Technical Field

[0001] The present invention relates to the technical fields of brain age prediction and image data processing, and specifically to a brain age prediction method, system, device and medium based on multi-modal information fusion. Background Art

[0002] With the rapid development of artificial intelligence technology, the cross-research in the fields of brain science and computer vision has gradually become a hot topic. Especially in the research of brain age prediction, it has promoted the in-depth development of medical intelligence. Brain age prediction is a technology that infers an individual's actual biological age by analyzing brain imaging data, and is widely used in the early screening of neurodegenerative diseases and the detection of abnormal brain development. Structural magnetic resonance imaging (sMRI) is the most commonly used data source in brain age prediction tasks because it can provide high-resolution three-dimensional anatomical images, showing the structural features of the brain, including key parameters such as gray matter volume, white matter volume, and cortical thickness. These features are closely related to the development and aging process of the brain and become important bases for predicting brain age.

[0003] Traditional image data processing methods face problems such as high dimensionality and a lot of noise. With the progress of neural network technology, convolutional neural networks (CNNs) have shown significant advantages in extracting important information from brain images. In the design of brain age prediction models, CNNs, due to their excellent local feature extraction ability, can efficiently extract information such as texture, edges, and structures of images from sMRI data, and through the stacking of multi-level convolutions, achieve a progressive extraction from low-level features to high-level semantic features. However, CNN models have certain limitations in processing the global information of images and cannot capture the complex connection relationships between brain regions.

[0004] Graph convolutional networks (GCNs) can capture the dependence relationships between different brain regions by modeling brain region nodes and their connections, enabling the model to not only extract local features of each brain region but also capture the global topological structure between brain regions, enhancing the accuracy of brain age prediction. However, GCNs mainly rely on the information aggregation of local neighbor nodes, which to a certain extent limits their ability to capture the relationships between distant brain regions. In addition, as the number of layers increases, GCNs may experience over-smoothing problems, resulting in the features of different brain regions becoming too similar, weakening the discrimination ability of the model, which may have a negative impact on the accuracy of brain age prediction. Summary of the Invention

[0005] The present invention provides a brain age prediction method, system, device and medium based on multi-modal information fusion.

[0006] The technical solution of the present invention is as follows:

[0007] A brain age prediction method based on multi-modal information fusion, comprising the following steps:

[0008] S1. Based on the acquired T1-weighted imaging, perform preprocessing on it to obtain the preprocessed image; after performing gray matter volume map segmentation and brain region division on it, obtain the radiomics features of each brain region, correspondingly generate the first brain region node feature matrix, perform weighted summation on the radiomics features of each brain region to obtain the feature value of each brain region, and construct the first adjacency feature matrix based on the correlation between the feature values;

[0009] S2. Sequentially perform several multi-scale enhancement processes on the preprocessed image to obtain several feature enhancement maps, and perform splicing processing on the several feature enhancement maps to obtain a brain image feature map;

[0010] The multi-scale enhancement process is as follows: The preprocessed image is respectively subjected to parallel convolution processing and addition fusion processing through several different convolution paths to obtain a multi-scale feature map; sequentially perform channel feature enhancement processing and spatial feature enhancement processing on the multi-scale feature map to obtain a feature enhancement map, and use the current feature enhancement map as the preprocessed image in the next multi-scale enhancement process; the number of channels in the several multi-scale enhancement processes increases with the increase of the number of processes;

[0011] S3. Sequentially perform binarization processing and sparsification processing on the first adjacency feature matrix, then perform aggregation of neighbor node features to obtain an aggregated feature, and perform parallel convolution processing and addition fusion processing on the aggregated feature through several different graph convolutions, and then sequentially perform the first feature compression processing, residual connection processing with the first brain region node feature matrix, and the second feature compression processing to obtain the second brain region node feature matrix;

[0012] After performing graph pooling processing on the first adjacency feature matrix, sequentially perform several edge convolution processes to obtain a feature-enhanced adjacency matrix, perform residual connection processing on the feature-enhanced adjacency matrix and the first adjacency feature matrix, and then sequentially perform row convolution processing and column convolution processing to obtain the second adjacency feature matrix;

[0013] Perform flattening processing and dimensionality reduction processing on the second brain region node feature matrix and the second adjacency feature matrix respectively, and then perform splicing to obtain a brain radiomics feature map;

[0014] S4. After performing convolution compression processing and flattening processing on the brain image feature map, perform splicing processing with the brain radiomics feature map in the channel dimension to obtain a preliminary fusion feature, perform linear transformation on the preliminary fusion feature and the brain radiomics feature map respectively, then perform attention mechanism processing to obtain a weighted fusion feature, and perform classification based on the weighted fusion feature to obtain a brain age prediction result.

[0015] Further, it also includes obtaining a gender label, encoding it to obtain a gender feature; performing compression processing and flattening processing on the weighted fusion feature to obtain a flattened fusion feature, concatenating the flattened fusion feature and the gender feature and then performing classification to obtain a brain age prediction result.

[0016] Specifically, several different convolutional paths in S2 are single convolutional processing, or a combination of several convolutional processings, or a combination of pooling processing and convolutional processing.

[0017] The preprocessed images are respectively subjected to parallel convolutional processing through several different convolutional paths. The specific operation is: the preprocessed images are respectively subjected to parallel convolutional processing through 4 different convolutional paths. The first convolutional path is single 1×1×1 convolutional processing, the second convolutional path is a combination of 1×1×1 convolutional processing and 3×3×3 convolutional processing, the third convolutional path is a combination of 1×1×1 convolutional processing and 5×5×5 convolutional processing, and the fourth convolutional path is a combination of 3×3 max pooling processing and 1×1×1 convolutional processing.

[0018] Specifically, after the first adjacency feature matrix in S3 is subjected to graph pooling processing, several edge convolutional processings are sequentially performed to obtain a feature-enhanced adjacency matrix. The edge convolutional processing is specifically: after the first adjacency feature matrix obtained by graph pooling processing is subjected to two-dimensional convolutional processing, it is expanded in the spatial dimension to obtain an expanded feature matrix, the expanded feature matrix is transposed to obtain a transposed feature matrix, an element-wise addition processing is performed based on the transposed feature matrix and the expanded feature matrix to obtain an associated feature matrix, global pooling processing is performed on the associated feature matrix in both the row and column directions to respectively obtain a row feature vector and a column feature vector, and a multiplication weighting processing is performed based on the row feature vector and the column feature vector and the first adjacency feature matrix to obtain a feature-enhanced adjacency matrix.

[0019] Further, in S2, the multi-scale feature map is sequentially subjected to channel feature enhancement processing and spatial feature enhancement processing. The channel feature enhancement processing is specifically: global pooling processing is performed on the multi-scale feature map by channel to obtain a global channel feature vector, after the global channel feature vector is subjected to per-channel adaptive weight assignment processing, it is multiplied with the feature maps of different sizes in the corresponding channels to obtain a channel feature-enhanced map.

[0020] Further, the spatial feature enhancement processing is specifically: global pooling processing is respectively performed on the channel feature-enhanced map in three directions in the spatial dimension to obtain a global spatial feature vector in each direction, convolutional and activation function processings are sequentially performed based on the global spatial feature vector to obtain three spatial attention weights, and the spatial attention weights are multiplied with the global spatial feature vectors in the corresponding directions to obtain a brain image feature map.

[0021] In step S3, after the residual connection processing of the feature-enhanced adjacency matrix, row convolution processing and column convolution processing are sequentially performed. Specifically, for the feature-enhanced adjacency feature matrix, a two-dimensional convolution of 1×the number of nodes is used to aggregate the information of all edges connected to a node to this node; the column convolution processing is to use a two-dimensional convolution of the number of nodes×1 to aggregate the node information aggregated by the row convolution processing to this node again.

[0022] The present invention also provides a brain age prediction system based on multi-modal information fusion, including:

[0023] A feature generation module: used to preprocess the acquired T1-weighted imaging to obtain a preprocessed image; after performing gray matter volume map segmentation and brain region division on it, obtain the radiomics features of each brain region, correspondingly generate a first brain region node feature matrix, perform weighted summation on the radiomics features of each brain region to obtain the feature value of each brain region, and construct a first adjacency feature matrix based on the correlation between the feature values;

[0024] A brain image feature extraction module: used to sequentially perform several multi-scale enhancement processes on the preprocessed image to obtain several feature-enhanced images, and perform splicing processing on the several feature-enhanced images to obtain a brain image feature map; the multi-scale enhancement process is: the preprocessed image is respectively subjected to parallel convolution processing and addition and fusion processing through several different convolution paths to obtain a multi-scale feature map; the multi-scale feature map is sequentially subjected to channel feature enhancement processing and spatial feature enhancement processing to obtain a feature-enhanced image, and the feature-enhanced image of this time is used as the preprocessed image in the next multi-scale enhancement process; the number of channels of the several multi-scale enhancement processes increases with the increase of the number of processing times;

[0025] A brain radiomics feature extraction module: used to sequentially perform binarization processing and sparsification processing on the first adjacency feature matrix, then perform aggregation of neighbor node features to obtain an aggregated feature, and perform parallel convolution processing and addition and fusion processing on the aggregated feature through several different graph convolutions, and then sequentially perform first feature compression processing, residual connection processing with the first brain region node feature matrix, and second feature compression processing to obtain a second brain region node feature matrix; after performing graph pooling processing on the first adjacency feature matrix, sequentially perform several edge convolution processes to obtain a feature-enhanced adjacency matrix, and perform row convolution processing and column convolution processing sequentially after the residual connection processing of the feature-enhanced adjacency matrix and the first adjacency feature matrix to obtain a second adjacency feature matrix; perform flattening processing and dimensionality reduction processing on the second brain region node feature matrix and the second adjacency feature matrix respectively, and then perform splicing to obtain a brain radiomics feature map;

[0026] Cross-modal feature fusion module: After the brain image feature map is processed by convolution compression and flattening, it is concatenated with the radiomics feature map of the brain image in the channel dimension to obtain preliminary fusion features. After performing linear transformations on the preliminary fusion features and the radiomics feature map of the brain image respectively, an attention mechanism is applied to obtain weighted fusion features, and classification is performed based on the weighted fusion features to obtain the brain age prediction result.

[0027] In addition, the present invention provides a brain age prediction device based on multi-modal information fusion, including a processor and a memory. When the processor executes the computer program stored in the memory, the above-mentioned brain age prediction method based on multi-modal information fusion is implemented.

[0028] The present invention also provides a computer-readable storage medium storing a computer program, which when executed by a processor can implement the steps of the above-mentioned brain age prediction method based on multi-modal information fusion.

[0029] The beneficial effects of the present invention are as follows:

[0030] 1. The present invention provides a brain age prediction method that fuses various different modal data such as brain image feature data, radiomics feature data of the brain image, and gender labels. For different forms of input data, different feature extraction methods are used, and through feature concatenation and cross-modal feature fusion, accurate prediction of the brain age is achieved. On the one hand, it compensates for the deficiencies of different data in extracting features, and on the other hand, it can explore the relationship between brain structure changes and age from multiple dimensions, providing a more accurate indicator basis for the early prevention and intervention of subsequent neurodegenerative diseases.

[0031] 2. The present invention performs multi-scale enhancement processing on the preprocessed image, combines multi-scale features and spatial attention information to obtain a more representative brain image feature map, retains important information in multiple scales, channels, and spatial dimensions, and can effectively handle complex medical image features.

[0032] 3. The present invention uses different feature extraction methods for the brain region node feature matrix and the adjacency feature matrix respectively to obtain the radiomics feature map of the brain image, that is, it considers the radiomics information of each brain region and the connection relationship between each brain region. The combination of the two provides a comprehensive radiomics information representation, which can provide more comprehensive brain information. Detailed implementation manners

[0033] The following will describe the exemplary embodiments of the present disclosure in more detail in combination with the solutions.

[0034] Embodiment

[0035] This embodiment provides a brain age prediction method based on multi-modal information fusion, including the following steps:

[0036] S1. Based on the acquired T1-weighted imaging, perform preprocessing on it to obtain a preprocessed image; after performing gray matter volume map segmentation and brain region division on it, obtain the radiomics features of each brain region, correspondingly generate a first brain region node feature matrix, perform weighted summation on the radiomics features of each brain region to obtain the feature value of each brain region, and construct a first adjacency feature matrix based on the correlation between the feature values.

[0037] Process the T1-weighted imaging (T1W1) data of structural magnetic resonance imaging (sMRI) using the standard preprocessing method of FSL5.10, including registration, brain extraction, and voxel intensity normalization operations.

[0038] First, perform linear registration on the original data, that is, perform an affine transformation on the image to roughly align it to the Montreal Neurological Institute (MNI) standard space. Then, based on the linear registration, further perform non-linear registration so that the image can be more precisely aligned with the standard space, and the non-linear registration takes into account local deformations, which can capture the subtle differences in brain structures between individuals; after the registration is completed, perform a brain extraction operation, that is, extract the brain tissue from the image, remove the skull, skin, and other non-brain tissues, remove interference factors, and focus more on the internal structural features of the brain.

[0039] Finally, perform voxel intensity normalization to standardize the preprocessed data into a three-dimensional volume with a certain voxel size. In this embodiment, the voxel size is 91×109×91, and the spatial resolution of the voxel is unified to isotropic 2mm 3 , so as to ensure that the spatial dimensions and resolutions of the images are consistent, facilitating subsequent feature extraction and analysis.

[0040] Segment the brain gray matter volume map from the T1WI data through the CAT12 tool and use ANTs for linear and non-linear registration to the MNI standard space. Subsequently, based on the automated anatomical labeling (AAL) template, divide the brain gray matter map into several brain regions. In this embodiment, 90 brain regions are specifically divided. For each brain region, calculate several radiomics features including intensity, texture, etc. The number of radiomics features is 25, and a 90×25 brain region node feature matrix is correspondingly generated as the first brain region node feature matrix, which is used to represent the features of each brain region.

[0041] Next, the radiomics features of each brain region are weighted and summed to obtain a feature value representing this brain region, and then the feature values of 90 brain regions are obtained. Then, the Pearson correlation coefficients are calculated for these 90 brain region feature values, so as to obtain an adjacency feature matrix (90×90) representing the correlation between each brain region as the first adjacency feature matrix. This matrix reflects the correlation of radiomics features between 90 brain regions (ROIs) and reveals the mutual relationship between different brain regions. The specific formula is as follows:

[0042] ,

[0043] where, is the correlation coefficient between the features of two brain regions, and are the feature values of two brain regions respectively, and are the average values of all feature values respectively.

[0044] After the processing of the above steps, the radiomics feature information of the brain can be obtained, including the individual features of brain regions and their connection relationships.

[0045] S2. The preprocessed images are sequentially subjected to several multi-scale enhancement processes to obtain several feature enhancement maps, and the several feature enhancement maps are subjected to stitching processing to obtain a brain image feature map;

[0046] The multi-scale enhancement process is as follows: The preprocessed images are respectively subjected to parallel convolution processing and addition and fusion processing through several different convolution paths to obtain multi-scale feature maps; The multi-scale feature maps are sequentially subjected to channel feature enhancement processing and spatial feature enhancement processing to obtain feature enhancement maps, and the feature enhancement maps of this time are used as the preprocessed images in the next multi-scale enhancement process; The number of channels in the several multi-scale enhancement processes increases with the increase of the number of processes.

[0047] When performing several multi-scale enhancement processes, the feature enhancement maps obtained each time are directly passed to perform the next multi-scale enhancement process, and all the obtained feature enhancement maps are subjected to stitching processing, which can capture low-level detail information and high-level abstract information and extract richer feature expressions, as shown in the following formula:

[0048] ,

[0049] where, is all the outputs from the first multi-scale enhancement process to the th multi-scale enhancement process; represents the th non-linear transformation function; represents the previous Concatenate all the outputs to form the input for the current multi-scale enhancement process.

[0050] In this embodiment, 5 times of multi-scale enhancement processing are set. The feature extraction method for each multi-scale enhancement processing is the same, and the channel numbers that increase successively are 8, 16, 32, 64, and 128 respectively. Multilevel image feature representations are obtained through the change of channel numbers.

[0051] In this embodiment, the convolutional path is a single convolutional process, or a combination of several convolutional processes, or a combination of pooling processing and convolutional processing.

[0052] For example, the preprocessed images are subjected to parallel convolutional processing through several different convolutional paths. The specific operation can be: the preprocessed images are subjected to parallel convolutional processing through 4 different convolutional paths. The first convolutional path is a single 1×1×1 convolutional process, which is used to reduce the channel dimension and retain local features; the second convolutional path is a combination of 1×1×1 convolutional processing and 3×3×3 convolutional processing, which is used to extract finer image details; the third convolutional path is a combination of 1×1×1 convolutional processing and 5×5×5 convolutional processing, which is used to capture a larger range of image features and enhance the perception ability of global information; the fourth convolutional path is a combination of 3×3 max pooling processing and 1×1×1 convolutional processing, which is used to reduce the image size and retain important feature information.

[0053] The above four different combinations are used to capture different scale information in the image and improve the capture ability of different features. These features are added after passing through the convolutional kernels and integrated into a new feature map, which is the multi-scale feature map. It contains image information at multiple scales and can better express the complex structure in the image.

[0054] ,

[0055] Among them, , , , respectively represent the image feature maps obtained after different branch convolutional or pooling operations. Finally, the results of the 4 branches are added to obtain the multi-scale feature map. .

[0056] Furthermore, channel feature enhancement processing is performed on the multi-scale feature map in sequence. The specific operation is: perform global pooling processing on the multi-scale feature map by channel to obtain the global channel feature vector. After performing per-channel adaptive weight assignment processing on the global channel feature vector, multiply it with the feature maps of different sizes in the corresponding channels to obtain the channel feature enhanced map.

[0057] Specifically, the multi-scale feature maps are sequentially subjected to channel feature enhancement processing, that is, global average pooling operation (AvgPooling) is performed on the multi-scale feature maps of each channel to generate a global channel description vector. This global description vector undergoes a per-channel adaptive weight assignment to dynamically adjust the weights of each channel, highlighting key channels and suppressing unimportant channel features. The obtained channel vector is multiplied by the original feature map to obtain a channel feature enhanced map in the channel dimension. The above operations can effectively enhance the recognition of image features and improve the focusing ability on channel dimension features. The channel dimension enhancement formula is as follows:

[0058] ,

[0059] where, is the channel feature enhanced map, is the sigmoid activation function, is the learnable weight matrix, is the multi-scale feature map, is the global average pooling operation, is the element-wise multiplication.

[0060] Furthermore, spatial feature enhancement processing is performed on the channel feature enhanced map. The specific operation is as follows: the channel feature enhanced map is globally pooled in three directions in the spatial dimension to obtain global spatial feature vectors in each direction. Based on the global spatial feature vectors, convolution and activation function processing are sequentially performed to obtain three spatial attention weights. The spatial attention weights are multiplied by the global spatial feature vectors in the corresponding directions to obtain the brain image feature map.

[0061] Specifically, spatial feature enhancement processing is performed on the feature map after channel enhancement. By performing feature processing on the three spatial dimensions of X, Y, and Z respectively, the feature expression at specific positions is further enhanced. First, the previously enhanced feature map is globally averaged in the X, Y, and Z directions to obtain global spatial information in each direction. Then, three independent spatial attention weights H, W, and D are generated through convolution and sigmoid operations. Finally, these three independent spatial attention weights are multiplied by the feature map to highlight the spatial features at specific positions in the three spatial dimensions. Through the attention mechanism in the X, Y, and Z directions, the spatial features can be enhanced more meticulously, ensuring that important information in space is not overlooked.

[0062] The spatial feature enhancement formula is as follows:

[0063] ,

[0064] where, is the brain image feature map after spatial feature enhancement, is the channel feature enhancement map, is the convolution operation in the X direction, is the global average pooling result in the X direction, is the convolution operation in the Y direction, is the global average pooling result in the Y direction, is the convolution operation in the Z direction, is the global average pooling result in the Z direction, is the sigmoid activation function, represents element-wise multiplication.

[0065] Finally, through the dual enhancement of the channel and spatial dimensions, the final brain image feature map is generated. This feature map retains important information in multiple scales, channels, and spatial dimensions and can effectively handle complex medical image features.

[0066] S3. After binarizing and sparsifying the first adjacency feature matrix in sequence, aggregate the features of neighbor nodes to obtain aggregated features. After parallel convolution processing and addition fusion processing of the aggregated features through several different graph convolutions, perform first feature compression processing, residual connection processing with the first brain region node feature matrix, and second feature compression processing in sequence to obtain the second brain region node feature matrix;

[0067] After performing graph pooling processing on the first adjacency feature matrix, perform edge convolution processing several times in sequence to obtain a feature-enhanced adjacency matrix. After performing residual connection processing on the feature-enhanced adjacency matrix and the first adjacency feature matrix, perform row convolution processing and column convolution processing in sequence to obtain the second adjacency feature matrix;

[0068] Flatten and reduce the dimensions of the second brain region node feature matrix and the second adjacency feature matrix respectively, and then splice them to obtain the radiomics feature map.

[0069] In this embodiment, the radiomics feature information is divided into 25 features of each brain region and the connection information between 90 brain regions, that is, the first brain region node feature matrix of 90×25 and the first adjacency feature matrix of 90×90. Different extraction methods are designed for these two radiomics features to extract brain information in all directions. Specifically, for the extraction of brain region node features, 90 brain regions are regarded as 90 nodes, and each node contains 25 feature information (such as texture, intensity, etc.), thus constituting a 90×25 brain region node feature matrix as the first brain region node feature matrix. Binarize the first adjacency feature matrix. By setting a threshold (threshold = 0.8), set the elements in the adjacency matrix greater than the threshold to 1 and the elements less than or equal to the threshold to 0, thereby obtaining the binarized adjacency matrix 。

[0070] To facilitate subsequent processing by the graph neural network, it is necessary to convert the binarized adjacency matrix into a sparse representation form, that is, convert the adjacency matrix into an edge list. This sparsification operation can be achieved through the dense_to_sparse function, and the specific operation is as follows:

[0071] ,

[0072] ,

[0073] where threshold = 0.8 is the set threshold, is the element in the i-th row and j-th column of the adjacency matrix, is the element of the binarized adjacency matrix; is the edge list of the sparse representation, represents an edge from node to node 。

[0074] After the sparsification process, the adjacency matrix is transformed from a dense matrix form into a sparse edge list representation, which is convenient for subsequent graph convolution operations. Determine all neighbor nodes that have a connection relationship with the current node according to the edge list, and aggregate the features of these neighbor nodes.

[0075] In this embodiment, a parallel graph convolution method is used, and multiple aggregation strategies are adopted to extract the diversity features of each brain region. For example, the average value, maximum value, and minimum value are used to extract the diversity features of each brain region, corresponding to three parallel graph convolutions. The three parallel graph convolution operations can respectively capture the dependence relationships between each brain region and different neighbor nodes, and then add and fuse these aggregated features, so that the node features integrate multi-level information. After the addition and fusion process, the dimension of the node features will increase.

[0076] To reduce the feature dimension, the node features are compressed by the first feature compression process (such as taking the average value), and the previous "multi-head features" are fused to obtain a more concentrated node feature representation. Suppose there is a node and its set of neighbor nodes, then the process of updating the feature representation of the node is as follows:

[0077] ,

[0078] where, represents the feature representation of node at the -th layer, is the trainable weight matrix, is the activation function RELU, is an aggregation function, which can be the maximum value, minimum value, average value, etc.; is a neighbor node at the layer features, is a neighbor node at the layer features, represents the set of neighbor nodes belonging to node v in the nodes u .

[0079] Furthermore, in order to prevent the problem of gradient disappearance in the deep network and improve the efficiency of information flow, a residual connection mechanism is introduced. The aggregated node features are residually connected with the first brain region node feature matrix to retain the original feature information and enhance the learning ability. Finally, the node features are subjected to a second feature compression process through a fully connected layer to further compress the feature dimension and obtain the second brain region node feature matrix, and these features will represent the characteristics of each brain region in the entire network.

[0080] The extraction of the features of the first adjacency feature matrix is specifically as follows. For the 90×90 first adjacency feature matrix, first perform graph pooling processing on it, screen out 50% of the more important connected nodes, remove the unimportant connections, simplify the matrix structure, and obtain a matrix, where \(R\) is the real number field, that is, the elements of this matrix belong to the set of real numbers.

[0081] Perform several edge convolution processes on the first adjacency feature matrix after graph pooling processing in sequence. The edge convolution process is specifically as follows: After performing two-dimensional convolution processing on the first adjacency feature matrix after graph pooling processing using a convolution kernel of size 1×the number of nodes, a feature matrix compressed in a certain dimension and with a changed size is obtained, then it is expanded in the spatial dimension to obtain an expanded feature matrix, the expanded feature matrix is transposed to obtain a transposed feature matrix, and an element-wise addition process is performed based on the transposed feature matrix and the expanded feature matrix to obtain an associated feature matrix. By introducing a transpose operation and adding the transposed matrix to the untransposed matrix, it is ensured that information can be transmitted bidirectionally between all nodes of the graph, enabling each node to receive positive and negative information from other nodes.

[0082] Then, global pooling is performed on the associated feature matrix in both the row and column directions. Among them, row-direction pooling aggregates the relationship information between each brain region and other brain regions, revealing the overall output correlation of this brain region; column-direction pooling aggregates the correlation information received by each brain region from other brain regions, showing the characteristics of this brain region as a receiver. The combination of row and column direction pooling can, based on two-way information flow, extract the most valuable feature representation between brain regions, which not only simplifies the feature dimension but also enhances the feature expression effect. After performing row global pooling and column global pooling, a row feature vector and a column feature vector are obtained respectively. Based on the row feature vector and the column feature vector, a multiplication weighting process is performed with the first adjacency feature matrix to obtain a feature-enhanced adjacency matrix. This not only enhances the expression ability of node features but also dynamically adjusts the weights between nodes, reduces information redundancy, improves computational efficiency, and enhances the diversity of feature aggregation. The specific operations are as follows:

[0083] ,

[0084] ,

[0085] Among them, is the associated feature matrix, is the feature-enhanced adjacency matrix, is to perform a 3×3 two-dimensional convolution on the matrix A , is the transpose operation, is the global average pooling operation, and perform operations in the row and column directions respectively, is the element-wise dot product.

[0086] In this embodiment, the edge convolution process is performed 3 times to gradually enhance the expression ability of feature information. After the above 3 times of edge convolution processing, a feature-enhanced adjacency matrix is obtained. After residual connection processing on the feature-enhanced adjacency matrix, a row convolution and a column convolution operation are introduced, and row convolution processing and column convolution processing are performed in sequence. Row convolution processing is to use a 1×number of nodes two-dimensional convolution to aggregate the information of all edges connected to a node to this node; column convolution processing is to use a number of nodes×1 two-dimensional convolution to aggregate the node information aggregated by row convolution processing to this node again, and finally a second adjacency feature matrix is obtained.

[0087] It should be further noted that, in order to retain the original information, for the 90×90 first adjacency feature matrix, after performing graph pooling processing on it, a 1×1 convolution is introduced for residual connection. At the end of the edge convolution processing, a residual connection operation is performed on the obtained feature-enhanced adjacency matrix, and a residual connection processing is performed with the first adjacency feature matrix after 1×1 convolution processing to retain the original information as much as possible.

[0088] Finally, the 90×25 second brain region node feature matrix and the 90×90 second adjacency feature matrix are respectively processed by a flattening operation and input into a fully connected layer for dimensionality reduction. Then, the two feature matrices are concatenated. Among them, the node features reflect the radiomics information of each brain region, while the adjacency matrix features reflect the connection relationships between brain regions. Combining the two can provide more comprehensive brain information.

[0089] S4. After the brain image feature map is processed by convolution compression and flattening, it is concatenated with the brain radiomics feature map in the channel dimension to obtain preliminary fusion features. After performing linear transformations on the preliminary fusion features and the brain radiomics feature map respectively, an attention mechanism is applied to obtain weighted fusion features, and classification is performed based on the weighted fusion features to obtain the brain age prediction result.

[0090] Specifically, the extracted brain image feature map contains three-dimensional spatial information (H×W×D), that is, the features of each voxel point. To adapt to cross-modal feature fusion, it is necessary to first compress the brain image feature map through a convolution operation, so that the channel dimension of the brain image feature map is adjusted to the same dimension as the brain radiomics feature map, ensuring that they can be concatenated in the same dimensional space. Then, the three-dimensional spatial information of the image is flattened into a one-dimensional sequence through a flattening operation to generate the compressed brain image features. For the brain radiomics feature map , in order to effectively fuse with the compressed brain image features, it is compressed through a fully connected layer, and then the compressed brain radiomics feature map is concatenated with the radiomics features in the channel dimension to generate preliminary fusion features , and the formula is as follows:

[0091] ,

[0092] where , is the batch size, representing the number of data samples input into the model at one time, is the dimension of the image features after convolution compression, is the compression dimension of the radiomics features; is the brain image feature map, is a 3×3×3 three-dimensional convolution, is the flattening operation; is the brain radiomics feature map, is the fully connected layer compression. This concatenation operation retains the information of both types of features, ensuring that features of different modalities are considered simultaneously.

[0093] To further capture the complex interaction between image features and radiomics features, a cross-modal attention mechanism is introduced. Specifically, the concatenated fusion features are mapped to a query vector Q through a linear transformation. The radiomics features are mapped to a key vector K and a value vector V through a linear transformation, and then processed using the Scaled Dot-Product Attention mechanism. The query vector Q is dotted with the key vector K to obtain the attention weights: where,

[0094] ,

[0095] ,

[0096] where, is the preliminary fusion feature, is the brain radiomics feature map is a linear transformation used to reduce the dimension; is the dimension of the key vector, is the attention mechanism processing. This attention mechanism dynamically weights the radiomics features according to the correlation between different modal features, enabling the fused features to effectively capture the interaction information between the image and radiomics features.

[0097] After performing linear transformations on the preliminary fusion feature and the brain radiomics feature map respectively, the attention mechanism is used to process them to obtain the weighted fusion feature. Based on the weighted fusion feature, compression processing and flattening processing are performed on it to obtain the flattened fusion feature. The flattened fusion feature is classified using a fully connected layer to obtain the brain age prediction result , and the formula is as follows:

[0098] ,

[0099] where, is the activation function RELU, and are the weights and biases of the fully connected layer respectively, is the flattening operation, is the compression of the fully connected layer, is the weighted fusion feature, is the gender feature, is the bias of the fully connected layer.

[0100] To further improve the prediction accuracy, in another embodiment, a gender label is introduced as auxiliary information. The gender label is one-hot encoded to obtain the encoded gender feature ( ). The weighted fusion feature is compressed and flattened to obtain the flattened fusion feature. After concatenating the flattened fusion feature and the gender feature, classification is performed to obtain the brain age prediction result, comprehensively considering various information including image, radiomics features, and gender labels; after concatenating the gender information and the fusion feature, fully connected processing is performed for regression prediction of the brain age, and the formula is as follows:

[0101] ,

[0102] where, is the activation function RELU, and are the weights and biases of the fully connected layer respectively, is the flattening operation, is the compression of the fully connected layer, is the weighted fusion feature, is the gender feature, is the bias of the fully connected layer.

[0103] The image features, radiomics features, and gender labels are effectively fused and finally used for the prediction of the brain age, improving the overall accuracy and expressive ability of the brain age prediction.

[0104] The present invention also provides a brain age prediction system based on multi-modal information fusion, including:

[0105] Feature generation module: used to preprocess the acquired T1-weighted imaging to obtain the preprocessed image; after performing gray matter volume map segmentation and brain region division on it, radiomics features of each brain region are obtained, and the first brain region node feature matrix is correspondingly generated. The radiomics features of each brain region are weighted and summed to obtain the feature value of each brain region, and based on the correlation between the feature values, the first adjacency feature matrix is constructed;

[0106] Brain image feature extraction module: used to sequentially perform several multi-scale enhancement processes on the preprocessed image to obtain several feature enhancement maps, and perform a splicing process on the several feature enhancement maps to obtain a brain image feature map; the multi-scale enhancement process is: the preprocessed image is respectively subjected to parallel convolution processing and addition and fusion processing through several different convolution paths to obtain a multi-scale feature map; the multi-scale feature map is sequentially subjected to channel feature enhancement processing and spatial feature enhancement processing to obtain a feature enhancement map, and the current feature enhancement map is used as the preprocessed image in the next multi-scale enhancement process; the number of channels in the several multi-scale enhancement processes increases with the increase of the number of processes;

[0107] Brain imaging omics feature extraction module: It is used to binarize and sparsify the first adjacency feature matrix in sequence, then aggregate the features of neighbor nodes to obtain aggregated features. After parallel convolution processing and addition fusion processing of the aggregated features through several different graph convolutions, first feature compression processing, residual connection processing with the first brain region node feature matrix, and second feature compression processing are performed in sequence to obtain the second brain region node feature matrix; after performing graph pooling processing on the first adjacency feature matrix, edge convolution processing is performed several times in sequence to obtain a feature-enhanced adjacency matrix. After performing residual connection processing on the feature-enhanced adjacency matrix and the first adjacency feature matrix, row convolution processing and column convolution processing are performed in sequence to obtain the second adjacency feature matrix; the second brain region node feature matrix and the second adjacency feature matrix are respectively flattened and dimension-reduced, and then spliced to obtain a brain imaging omics feature map;

[0108] Cross-modal feature fusion module: It is used to perform convolution compression processing and flattening processing on the brain image feature map, then splice it with the brain imaging omics feature map in the channel dimension to obtain preliminary fusion features. After performing linear transformation on the preliminary fusion features and the brain imaging omics feature map respectively, attention mechanism processing is performed to obtain weighted fusion features, and classification is performed based on the weighted fusion features to obtain a brain age prediction result.

[0109] In addition, the present invention also provides a brain age prediction device based on multi-modal information fusion, including a processor and a memory. Among them, when the processor executes the computer program stored in the memory, the brain age prediction method based on multi-modal information fusion as described above is implemented.

[0110] Finally, the present invention provides a computer-readable storage medium storing a computer program, and when this program is executed by a processor, the brain age prediction method based on multi-modal information fusion as described above can be implemented.

[0111] Experimental content

[0112] Experimental settings: The dataset is randomly divided into a training set, a validation set, and a test set according to the ratio of 75%, 15%, and 10%. The experimental development environment used is pytorch1.9.0. On the NVIDIA RTX 2080 graphics processor, a total of 100 rounds were carried out during the training period. The Adam optimizer was used, the initial learning rate was 1e-3, and the batch size was set to 2. The brain age prediction method based on multi-modal information fusion of the present application was trained on the NVIDIA RTX 2080 graphics processor for about 6 hours. The experimental environment and specific settings are shown in Table 1, and the brain age prediction results are shown in Table 2.

[0113] Table

[0114]

[0115] Table 2

[0116]

[0117] The present invention provides a brain age prediction method, system, device and medium based on multi-modal information fusion. For different forms of input information, the method designs different feature extraction methods. For image information, a multi-scale and multi-dimensional feature extraction and enhancement method based on dense connection is designed, which can capture multi-scale features and avoid the limitations of traditional convolutional neural networks (CNNs); for radiomics information, a parallel graph convolutional processing method and a unique combination processing of edge convolution and row-column convolution are designed, which are suitable for extracting complex graph structure information in non-Euclidean spaces and make up for the deficiencies of traditional Euclidean space methods in graph structure information extraction. The present invention performs multi-modal feature fusion through feature splicing and cross-modal attention mechanisms, and finally realizes accurate prediction of brain age. The present invention proposes a strategy of combining medical image data focusing on spatial information with radiomics features focusing on quantitative features and statistical information, fuses multi-modal data such as structural information in images and quantitative information in radiomics, and applies it to the field of brain age prediction. The image information and radiomics information complement each other, explore the relationship between brain structure information and brain age more comprehensively from multiple angles, and lay a foundation for subsequent research on brain structure changes.

Claims

1. A brain age prediction method based on multimodal information fusion, characterized in that, It includes the following steps: S1. Based on the acquired T1-weighted imaging, perform preprocessing on it to obtain a preprocessed image; After performing gray matter volume map segmentation and brain region division on it, obtain the radiomics features of each brain region, correspondingly generate the first brain region node feature matrix, perform weighted summation on the radiomics features of each brain region to obtain the eigenvalue of each brain region, and construct the first adjacency feature matrix based on the correlation between the eigenvalues; S2. Sequentially perform several multi-scale enhancement processes on the preprocessed image to obtain several feature-enhanced images, and perform splicing processing on the several feature-enhanced images to obtain a brain image feature map; The multi-scale enhancement process is as follows: respectively perform parallel convolution processing and addition and fusion processing on the preprocessed image through several different convolution paths to obtain a multi-scale feature map; sequentially perform channel feature enhancement processing and spatial feature enhancement processing on the multi-scale feature map to obtain a feature-enhanced image, and use the current feature-enhanced image as the preprocessed image in the next multi-scale enhancement process; the number of channels in the several multi-scale enhancement processes increases with the increase of the number of processes; S3. After sequentially performing binarization processing and sparsification processing on the first adjacency feature matrix, perform aggregation of neighbor node features to obtain an aggregated feature, and perform parallel convolution processing and addition and fusion processing on the aggregated feature through several different graph convolutions, and then sequentially perform the first feature compression processing, residual connection processing with the first brain region node feature matrix, and the second feature compression processing to obtain the second brain region node feature matrix; After performing graph pooling processing on the first adjacency feature matrix, sequentially perform several edge convolution processes to obtain a feature-enhanced adjacency matrix, and perform residual connection processing on the feature-enhanced adjacency matrix and the first adjacency feature matrix, and then sequentially perform row convolution processing and column convolution processing to obtain the second adjacency feature matrix; Perform flattening processing and dimensionality reduction processing on the second brain region node feature matrix and the second adjacency feature matrix respectively, and then perform splicing to obtain a brain radiomics feature map; S4. After performing convolution compression processing and flattening processing on the brain image feature map, perform splicing processing with the brain radiomics feature map in the channel dimension to obtain a preliminary fusion feature, perform linear transformation on the preliminary fusion feature and the brain radiomics feature map respectively, and then perform attention mechanism processing to obtain a weighted fusion feature, and perform classification based on the weighted fusion feature to obtain a brain age prediction result.

2. The brain age prediction method based on multimodal information fusion according to claim 1, wherein It also includes obtaining a gender label, encoding it to obtain a gender feature; performing compression processing and flattening processing on the weighted fusion feature to obtain a flattened fusion feature, splicing the flattened fusion feature and the gender feature and then performing classification to obtain a brain age prediction result.

3. The brain age prediction method based on multi-modal information fusion according to claim 1, wherein The several different convolution paths in S2 are single convolution processing, or a combination of several convolution processes, or a combination of pooling processing and convolution processing; The pre-processed image is subjected to parallel convolution processing through a number of different convolution paths. The specific operation is as follows: The pre-processed image is subjected to parallel convolution processing through 4 different convolution paths. The first convolution path is a single 1×1×1 convolution processing. The second convolution path is a combination of 1×1×1 convolution processing and 3×3×3 convolution processing. The third convolution path is a combination of 1×1×1 convolution processing and 5×5×5 convolution processing. The fourth convolution path is a combination of 3×3 max pooling processing and 1×1×1 convolution processing.

4. The brain age prediction method based on multi-modal information fusion according to claim 1, wherein In S3, after the first adjacent feature matrix is subjected to graph pooling processing, several edge convolution processes are performed in sequence to obtain a feature-enhanced adjacent matrix. The specific edge convolution process is as follows: After the first adjacent feature matrix obtained by graph pooling processing is subjected to two-dimensional convolution processing, it is extended in the spatial dimension to obtain an extended feature matrix. The extended feature matrix is transposed to obtain a transposed feature matrix. Element-wise addition processing is performed based on the transposed feature matrix and the extended feature matrix to obtain an associated feature matrix. Global pooling processing is performed on the associated feature matrix in both the row and column directions to obtain a row feature vector and a column feature vector respectively. Multiplication weighting processing is performed based on the row feature vector, the column feature vector, and the first adjacent feature matrix to obtain a feature-enhanced adjacent matrix.

5. The brain age prediction method based on multi-modal information fusion according to claim 1, wherein, In S2, channel feature enhancement processing and spatial feature enhancement processing are performed on the multi-scale feature map in sequence. The specific operation of the channel feature enhancement processing is as follows: Global pooling processing is performed on the multi-scale feature map by channel to obtain a global channel feature vector. After the global channel feature vector is subjected to per-channel adaptive weight assignment processing, it is multiplied with the feature maps of different sizes in the corresponding channels to obtain a channel feature-enhanced map.

6. The brain age prediction method based on multi-modal information fusion according to claim 5, characterized in that, The specific operation of the spatial feature enhancement processing is as follows: Global pooling processing is performed on the channel feature-enhanced map in three directions in the spatial dimension to obtain a global spatial feature vector for each direction. Based on the global spatial feature vector, convolution and activation function processing are performed in sequence to obtain three spatial attention weights. The spatial attention weights are multiplied with the global spatial feature vectors in the corresponding directions to obtain a brain image feature map.

7. The brain age prediction method based on multimodal information fusion according to claim 1, wherein In S3, after residual connection processing is performed on the feature-enhanced adjacent matrix, row convolution processing and column convolution processing are performed in sequence. Specifically: For the feature-enhanced adjacent feature matrix, a two-dimensional convolution of 1×number of nodes is used to aggregate the information of all edges connected to a node to that node; The column convolution processing is to use a two-dimensional convolution of number of nodes×1 to aggregate the node information aggregated by the row convolution processing to that node again.

8. A brain age prediction system based on multi-modal information fusion, characterized in that, Including: Feature generation module: used to pre-process the acquired T1-weighted imaging to obtain a pre-processed image; After performing gray matter volume map segmentation and brain region division on it, the radiomics features of each brain region are obtained, and a first brain region node feature matrix is correspondingly generated. The radiomics features of each brain region are weighted and summed to obtain the feature value of each brain region. Based on the correlation between the feature values, a first adjacent feature matrix is constructed; Brain image feature extraction module: It is used to sequentially perform multi-scale enhancement processing on the preprocessed image for several times to obtain several feature-enhanced images, and splice the several feature-enhanced images to obtain a brain image feature map; the multi-scale enhancement processing is as follows: the preprocessed image is respectively subjected to parallel convolution processing and addition and fusion processing through several different convolution paths to obtain a multi-scale feature map; the multi-scale feature map is sequentially subjected to channel feature enhancement processing and spatial feature enhancement processing to obtain a feature-enhanced image, and the feature-enhanced image of this time is used as the preprocessed image in the next multi-scale enhancement processing; the number of channels in the several multi-scale enhancement processings increases with the increase of the number of processings; Brain radiomics feature extraction module: It is used to sequentially perform binarization processing and sparsification processing on the first adjacency feature matrix, then aggregate the features of neighbor nodes to obtain an aggregated feature, and perform parallel convolution processing and addition and fusion processing on the aggregated feature through several different graph convolutions, and then sequentially perform first feature compression processing, residual connection processing with the first brain region node feature matrix, and second feature compression processing to obtain a second brain region node feature matrix; after performing graph pooling processing on the first adjacency feature matrix, sequentially perform several edge convolution processings to obtain a feature-enhanced adjacency matrix, and perform residual connection processing on the feature-enhanced adjacency matrix and the first adjacency feature matrix, and then perform row convolution processing and column convolution processing to obtain a second adjacency feature matrix; After flattening processing and dimensionality reduction processing are respectively performed on the second brain region node feature matrix and the second adjacency feature matrix and then spliced, a brain radiomics feature map is obtained; Cross-modal feature fusion module: It is used to perform convolution compression processing and flattening processing on the brain image feature map, and then splice it with the brain radiomics feature map in the channel dimension to obtain a preliminary fusion feature. After performing linear transformation on the preliminary fusion feature and the brain radiomics feature map respectively, perform attention mechanism processing to obtain a weighted fusion feature, and perform classification based on the weighted fusion feature to obtain a brain age prediction result.

9. A brain age prediction device based on multi-modal information fusion, characterized in that, It includes a processor and a memory. Among them, when the processor executes the computer program stored in the memory, it implements the brain age prediction method based on multi-modal information fusion according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It is used to store a computer program. Among them, when the computer program is executed by the processor, it implements the brain age prediction method based on multi-modal information fusion according to any one of claims 1-7.

Images