Brain age prediction method, system and device based on multi-modal information fusion and medium

Abstract

The invention relates to the technical field of brain age prediction and image data processing, in particular to a brain age prediction method, system and device based on multi-modal information fusion and a medium. Multi-scale enhancement processing is carried out on a preprocessed image, multi-scale features and space attention information are combined, a brain image feature map is obtained, and a brain age prediction result is obtained; different feature extraction modes are respectively used for the brain region node feature matrix and the adjacent feature matrix, a plurality of different image convolutions are used for carrying out parallel convolution processing to extract brain region node features, an edge convolution processing and row convolution column convolution processing combined mode is used for extracting adjacent features, a brain image omics feature map is obtained after splicing, and the brain region node features are extracted. And further performing feature splicing and cross-modal feature fusion processing on the brain image feature map and the brain radiomics feature map to realize accurate prediction of the brain age.

Description

Technical Field

[0001] The present invention relates to the technical field of brain age prediction and image data processing, and in particular to a brain age prediction method, system, device and medium based on multimodal information fusion. Background Art

[0002] With the rapid development of artificial intelligence technology, cross-research in the fields of brain science and computer vision has gradually become a hot topic, especially in the research of brain age prediction, which has promoted the in-depth development of medical intelligence. Brain age prediction is a technology that infers the actual biological age of an individual by analyzing brain imaging data. It is widely used in early screening of neurodegenerative diseases and detection of abnormal brain development. Structural magnetic resonance imaging (sMRI) is the most commonly used data source in brain age prediction tasks because it can provide high-resolution three-dimensional anatomical images and show the structural characteristics of the brain, including key parameters such as gray matter volume, white matter volume, and cortical thickness. These characteristics are closely related to the development and aging process of the brain and have become an important basis for predicting brain age.

[0003] Traditional image data processing methods face problems such as high dimensionality and high noise. With the advancement of neural network technology, convolutional neural networks (CNNs) have shown significant advantages in extracting important information from brain images. In the design of models for brain age prediction, CNNs can efficiently extract image texture, edge, and structure information from sMRI data due to their excellent local feature extraction capabilities, and achieve progressive extraction from low-level features to high-level semantic features through the stacking of multi-level convolutions. However, the CNN model has certain limitations in processing the global information of images and cannot capture the complex connection relationships between brain regions.

[0004] Graph convolutional networks (GCNs) can capture the dependencies between different brain regions by modeling brain region nodes and their interconnections, allowing the model to not only extract local features of each brain region, but also capture the global topological structure between brain regions, thereby enhancing the accuracy of brain age prediction. However, GCNs mainly rely on information aggregation of local neighbor nodes, which to some extent limits their ability to capture relationships between distant brain regions. In addition, as the number of layers increases, GCNs may have over-smoothing problems, causing the features of different brain regions to become too similar, weakening the model's ability to distinguish, which may have a negative impact on the accuracy of brain age prediction. Summary of the invention

[0005] The present invention provides a brain age prediction method, system, device and medium based on multimodal information fusion.

[0006] The technical solution of the present invention is as follows: A brain age prediction method based on multimodal information fusion comprises the following steps: S1. Based on the acquired T1 weighted imaging, preprocessing is performed to obtain a preprocessed image; after gray matter volume segmentation and brain region division, the imaging genomics features of each brain region are obtained, and a first brain region node feature matrix is ​​generated accordingly, and the imaging genomics features of each brain region are weighted summed to obtain a feature value of each brain region, and a first adjacency feature matrix is ​​constructed based on the correlation between the feature values; S2, subjecting the preprocessed image to multiple multi-scale enhancement processes in sequence to obtain multiple feature enhancement maps, and performing splicing processing on the multiple feature enhancement maps to obtain a brain image feature map; The multi-scale enhancement processing is as follows: the pre-processed image is subjected to parallel convolution processing and addition fusion processing through several different convolution paths respectively to obtain a multi-scale feature map; the multi-scale feature map is subjected to channel feature enhancement processing and spatial feature enhancement processing in sequence to obtain a feature enhancement map, and the feature enhancement map of this time is used as the pre-processed image in the next multi-scale enhancement processing; the number of channels of the multi-scale enhancement processing increases with the increase of the number of processing times; S3, after binarizing and sparse processing the first adjacency feature matrix in sequence, aggregate the neighbor node features to obtain aggregated features, perform parallel convolution processing and addition fusion processing on the aggregated features through several different graph convolutions, and then perform first feature compression processing, residual connection processing with the first brain region node feature matrix, and second feature compression processing in sequence to obtain a second brain region node feature matrix; After performing graph pooling processing on the first adjacency feature matrix, several edge convolution processing is performed in sequence to obtain a feature enhanced adjacency matrix, and after residual connection processing on the feature enhanced adjacency matrix and the first adjacency feature matrix, row convolution processing and column convolution processing are performed in sequence to obtain a second adjacency feature matrix; The node feature matrix of the second brain region and the second adjacent feature matrix were flattened and dimensionally reduced, and then spliced ​​to obtain the brain imaging omics feature map; S4. After the brain image feature map is processed by convolution compression and flattening, it is spliced ​​with the brain imaging genomics feature map in the channel dimension to obtain preliminary fusion features. The preliminary fusion features and the brain imaging genomics feature map are linearly transformed respectively, and then processed by the attention mechanism to obtain weighted fusion features. Classification is performed based on the weighted fusion features to obtain brain age prediction results.

[0007] Furthermore, it also includes obtaining a gender label, encoding it, and obtaining a gender feature; compressing and flattening the weighted fusion feature to obtain a flattened fusion feature, splicing the flattened fusion feature and the gender feature and classifying them to obtain a brain age prediction result.

[0008] Specifically, the several different convolution paths in S2 are a single convolution process, or a combination of several convolution processes, or a combination of pooling process and convolution process.

[0009] The preprocessed image is subjected to parallel convolution processing through several different convolution paths. The specific operation is as follows: the preprocessed image is subjected to parallel convolution processing through 4 different convolution paths. The first convolution path is a single 1×1×1 convolution processing, the second convolution path is a combination of 1×1×1 convolution processing and 3×3×3 convolution processing, the third convolution path is a combination of 1×1×1 convolution processing and 5×5×5 convolution processing, and the fourth convolution path is a combination of 3×3 maximum pooling processing and 1×1×1 convolution processing.

[0010] Specifically, in S3, after the first adjacency feature matrix is ​​subjected to graph pooling processing, several edge convolution processing is performed in sequence to obtain a feature enhanced adjacency matrix, and the edge convolution processing is specifically as follows: after the first adjacency feature matrix after graph pooling processing is subjected to two-dimensional convolution processing, it is expanded in the spatial dimension to obtain an expanded feature matrix, the expanded feature matrix is ​​transposed to obtain a transposed feature matrix, element-by-element addition processing is performed based on the transposed feature matrix and the expanded feature matrix to obtain a correlation feature matrix, global pooling processing is performed on the correlation feature matrix in both row and column directions to obtain row feature vectors and column feature vectors correspondingly, and weighted processing is performed based on multiplication of the row feature vectors and the column feature vectors with the first adjacency feature matrix to obtain a feature enhanced adjacency matrix.

[0011] Furthermore, in S2, channel feature enhancement processing and spatial feature enhancement processing are performed on the multi-scale feature map in sequence. The specific operation of the channel feature enhancement processing is: global pooling processing is performed on the multi-scale feature map by channel to obtain a global channel feature vector, and the global channel feature vector is subjected to channel-by-channel adaptive weight allocation processing, and then multiplied with feature maps of different sizes of the corresponding channel to obtain a channel feature enhancement map.

[0012] Furthermore, the specific operation of the spatial feature enhancement processing is: globally pooling the channel feature enhancement map in three directions of the spatial dimension to obtain the global spatial feature vector in each direction, performing convolution and activation function processing in sequence based on the global spatial feature vector to obtain three spatial attention weights, multiplying the spatial attention weights with the global spatial feature vector in the corresponding direction to obtain a brain image feature map.

[0013] In the S3, after the residual connection processing of the feature enhanced adjacency matrix, row convolution processing and column convolution processing are performed in sequence, specifically: a two-dimensional convolution of 1× the number of nodes is used for the feature enhanced adjacency feature matrix to aggregate the information of all edges connected to a node to the node; the column convolution processing is a two-dimensional convolution using the number of nodes × 1, and the node information aggregated by the row convolution processing is aggregated to the node again.

[0014] The present invention also provides a brain age prediction system based on multimodal information fusion, comprising: Feature generation module: used to preprocess the acquired T1-weighted imaging to obtain a preprocessed image; after gray matter volume segmentation and brain region division, the imaging features of each brain region are obtained, and the first brain region node feature matrix is ​​generated accordingly; the imaging features of each brain region are weighted summed to obtain the feature value of each brain region; and the first adjacency feature matrix is ​​constructed based on the correlation between the feature values; Brain image feature extraction module: used to sequentially subject the preprocessed image to multiple multi-scale enhancement processing to obtain multiple feature enhancement maps, and to splice the multiple feature enhancement maps to obtain a brain image feature map; the multiple multi-scale enhancement processing is: subjecting the preprocessed image to multiple different convolution paths for parallel convolution processing and addition fusion processing to obtain a multi-scale feature map; sequentially subjecting the multi-scale feature map to channel feature enhancement processing and spatial feature enhancement processing to obtain a feature enhancement map, and use the feature enhancement map of this time as the preprocessed image in the next multiple multi-scale enhancement processing; the number of channels of the multiple multi-scale enhancement processing increases with the increase of the number of processing times; Brain imaging omics feature extraction module: used to perform binarization and sparse processing on the first adjacency feature matrix in sequence, aggregate neighbor node features to obtain aggregated features, perform parallel convolution processing and addition fusion processing on the aggregated features through several different graph convolutions, and then perform first feature compression processing, residual connection processing with the first brain region node feature matrix, and second feature compression processing in sequence to obtain the second brain region node feature matrix; perform graph pooling processing on the first adjacency feature matrix, and then perform several edge convolution processing in sequence to obtain a feature enhanced adjacency matrix, perform row convolution processing and column convolution processing on the feature enhanced adjacency matrix and the first adjacency feature matrix after residual connection processing to obtain the second adjacency feature matrix; flatten and reduce the dimension of the second brain region node feature matrix and the second adjacency feature matrix, and then splice them to obtain a brain imaging omics feature map; Cross-modal feature fusion module: It is used to concatenate the brain image feature map with the brain imaging genomics feature map in the channel dimension after convolution compression and flattening to obtain preliminary fusion features. After linear transformation of the preliminary fusion features and the brain imaging genomics feature map, attention mechanism processing is performed to obtain weighted fusion features. Classification is performed based on the weighted fusion features to obtain brain age prediction results.

[0015] In addition, the present invention provides a brain age prediction device based on multimodal information fusion, comprising a processor and a memory, wherein the processor implements the brain age prediction method based on multimodal information fusion as described above when executing a computer program stored in the memory.

[0016] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps of the brain age prediction method based on multimodal information fusion as described above.

[0017] The beneficial effects of the present invention are: 1. The present invention provides a brain age prediction method that integrates brain image feature data, brain imaging genomics feature data and gender label data of multiple different modalities. Different feature extraction methods are used for different forms of input data, and feature splicing and cross-modal feature fusion are used to achieve accurate prediction of brain age. On the one hand, it complements the deficiencies of different data extraction features, and on the other hand, it can explore the relationship between brain structure changes and age from multiple dimensions, providing a more accurate indicator basis for the early prevention and intervention of subsequent neurodegenerative diseases.

[0018] 2. The present invention performs multi-scale enhancement processing on the pre-processed image, combines multi-scale features and spatial attention information, and obtains a more representative brain image feature map, retains important information in multi-scale, channel and spatial dimensions, and can effectively cope with complex medical image features.

[0019] 3. The present invention uses different feature extraction methods for the brain region node feature matrix and the adjacency feature matrix to obtain a brain imaging genomics feature map, which takes into account the imaging genomics information of each brain region and the connection relationship between each brain region. The combination of the two obtains a comprehensive imaging genomics information representation, which can provide more comprehensive brain information. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present disclosure will be described in more detail below in conjunction with the embodiments.

[0021] Example This embodiment provides a brain age prediction method based on multimodal information fusion, comprising the following steps: S1. Based on the acquired T1 weighted imaging, the imaging is preprocessed to obtain a preprocessed image; after gray matter volume segmentation and brain region division, the imaging genomics features of each brain region are obtained, and a first brain region node feature matrix is ​​generated accordingly. The imaging genomics features of each brain region are weighted summed to obtain a eigenvalue of each brain region, and a first adjacency feature matrix is ​​constructed based on the correlation between the eigenvalues.

[0022] The T1-weighted imaging (T1W1) data of structural magnetic resonance imaging (sMRI) were processed using the standard preprocessing method of FSL5.10, including registration, brain extraction, and voxel intensity normalization operations.

[0023] First, linear registration is used for the original data, that is, an affine transformation is performed on the image to roughly align it to the Montreal Neurological Institute (MNI) standard space. Then, based on the linear registration, nonlinear registration is further performed so that the image can be more accurately aligned with the standard space. In addition, nonlinear registration takes local deformation into account, which can capture subtle differences in brain structure between individuals. After the registration is completed, a brain extraction operation is performed, that is, the brain tissue is extracted from the image, the skull, skin and other non-brain tissues are removed, interference factors are removed, and more focus is placed on the internal structural characteristics of the brain.

[0024] Finally, voxel intensity normalization is performed to normalize the preprocessed data into a three-dimensional volume with a certain voxel size. In this embodiment, the voxel size is 91×109×91, and the spatial resolution of the voxels is unified to an isotropic 2 mm 3 , thereby ensuring the consistency of the spatial size and resolution of the image, facilitating subsequent feature extraction and analysis.

[0025] The T1WI data was segmented into a gray matter volume map using the CAT12 tool, and ANTs were used for linear and nonlinear registration to the MNI standard space. Subsequently, the gray matter map was divided into several brain regions based on the automatic anatomical labeling (AAL) template. In this embodiment, 90 brain regions were specifically divided. For each brain region, several imaging genomics features including intensity, texture, etc. were calculated. The number of imaging genomics features was 25, and a 90×25 brain region node feature matrix was generated as the first brain region node feature matrix, which was used to represent the characteristics of each brain region.

[0026] Next, the weighted sum of the imaging genomics features of each brain region is used to obtain a feature value representing this brain region, and then the feature values ​​of 90 brain regions are obtained. Then, the Pearson correlation coefficient is calculated for these 90 brain region feature values, so as to obtain the adjacency feature matrix (90×90) representing the correlation between each brain region as the first adjacency feature matrix. This matrix reflects the correlation of imaging genomics features between 90 brain regions (ROIs) and reveals the relationship between different brain regions. The specific formula is as follows: , in, is the correlation coefficient between the features of two brain regions, and are the characteristic values ​​of the two brain regions, and are the average values ​​of all eigenvalues.

[0027] After the above steps, the imaging genomics feature information of the brain can be obtained, including the individual characteristics of brain regions and their connection relationships.

[0028] S2, subjecting the preprocessed image to multiple multi-scale enhancement processes in sequence to obtain multiple feature enhancement maps, and performing splicing processing on the multiple feature enhancement maps to obtain a brain image feature map; The multi-scale enhancement processing is as follows: the pre-processed image is subjected to parallel convolution processing and addition fusion processing through several different convolution paths to obtain a multi-scale feature map; the multi-scale feature map is subjected to channel feature enhancement processing and spatial feature enhancement processing in sequence to obtain a feature enhancement map, and the feature enhancement map of this time is used as the pre-processed image in the next multi-scale enhancement processing; the number of channels of the several multi-scale enhancement processing increases with the increase of the number of processing times.

[0029] When performing several multi-scale enhancement processes, the feature enhancement map obtained each time is directly passed to the next multi-scale enhancement process, and all the obtained feature enhancement maps are spliced, which can capture low-level detail information and high-level abstract information, and extract richer feature expressions, as shown in the following formula: , in, From the first multi-scale enhancement process to the All outputs of the multiscale enhancement process; Indicates The nonlinear transformation function of the second order; Indicates that the All the outputs are concatenated to form the input of the current multi-scale enhancement process.

[0030] In this embodiment, five multi-scale enhancement processes are set. The feature extraction method of each multi-scale enhancement process is consistent, and a successively increasing number of channels is used, which are 8, 16, 32, 64 and 128 respectively. Multi-level image feature representation is obtained by changing the number of channels.

[0031] In this embodiment, the convolution path is a single convolution process, or a combination of several convolution processes, or a combination of pooling process and convolution process.

[0032] For example, the preprocessed image is subjected to parallel convolution processing through several different convolution paths. The specific operation can be: the preprocessed image is subjected to parallel convolution processing through 4 different convolution paths. The first convolution path is a single 1×1×1 convolution processing, which is used to reduce the channel dimension and retain local features; the second convolution path is a combination of 1×1×1 convolution processing and 3×3×3 convolution processing, which is used to extract finer image details; the third convolution path is a combination of 1×1×1 convolution processing and 5×5×5 convolution processing, which is used to capture a larger range of image features and enhance the perception of global information; the fourth convolution path is a combination of 3×3 maximum pooling processing and 1×1×1 convolution processing, which is used to reduce the image size and retain important feature information.

[0033] The above four different combinations are used to capture information of different scales in the image and improve the ability to capture different features. These features are added after the convolution kernel and integrated into a new feature map, which is a multi-scale feature map. It contains image information at multiple scales and can better express the complex structure in the image.

[0034] , in, , , , They represent the image feature maps obtained after different branch convolution or pooling operations, and finally the four branch results are added together to obtain a multi-scale feature map .

[0035] Furthermore, channel feature enhancement processing is performed on the multi-scale feature map in sequence. The specific operation is as follows: global pooling processing is performed on the multi-scale feature map by channel to obtain a global channel feature vector, and the global channel feature vector is adaptively weighted channel by channel, and then multiplied with feature maps of different sizes of the corresponding channel to obtain a channel feature enhancement map.

[0036] Specifically, the multi-scale feature maps are processed with channel feature enhancement in sequence, that is, a global average pooling operation (AvgPooling) is performed on the multi-scale feature maps of each channel to generate a global channel description vector. This global description vector is used to dynamically adjust the weight of each channel through a channel-by-channel adaptive weight allocation, highlighting key channels and suppressing unimportant channel features. The obtained channel vector is multiplied with the original feature map to obtain a channel feature enhancement map in the channel dimension. The above operation can effectively enhance the recognition of image features and improve the ability to focus on channel dimension features. The channel dimension enhancement formula is as follows: , in, is the channel feature enhancement map, is the sigmoid activation function, is a learnable weight matrix, is a multi-scale feature map, is the global average pooling operation, is the element-wise dot product.

[0037] Furthermore, spatial feature enhancement processing is performed on the channel feature enhancement map. The specific operations are as follows: globally pooling processing is performed on the channel feature enhancement map in three directions of the spatial dimension to obtain the global spatial feature vector in each direction; convolution and activation function processing are performed in sequence based on the global spatial feature vector to obtain three spatial attention weights; the spatial attention weights are multiplied with the global spatial feature vector in the corresponding direction to obtain the brain image feature map.

[0038] Specifically, the spatial feature enhancement processing is performed on the feature map after channel enhancement, and the feature expression of the specific position is further enhanced by performing feature processing on the three spatial dimensions of X, Y, and Z respectively. First, the previously enhanced feature map is globally averaged pooled in the three directions of X, Y, and Z to obtain the global spatial information in each direction, and then three independent spatial attention weights H, W, and D are generated through convolution and sigmoid operations. Finally, these three independent spatial attention weights are multiplied with the feature map to highlight the spatial features of specific positions in the three spatial dimensions. Through the attention mechanism in the three directions of X, Y, and Z, the spatial features can be enhanced more finely to ensure that important information in space is not ignored.

[0039] The spatial feature enhancement formula is as follows: , in, is the brain image feature map after spatial feature enhancement, is the channel feature enhancement map, is the convolution operation in the X direction, is the global average pooling result in the X direction, is the convolution operation in the Y direction, is the global average pooling result in the Y direction, is the convolution operation in the Z direction, is the global average pooling result in the Z direction, is the sigmoid activation function, Represents element-wise dot product.

[0040] Finally, after dual enhancement of channel and spatial dimensions, the final brain image feature map is generated. This feature map retains important information in multiple scales, channels and spatial dimensions, and can effectively cope with complex medical image features.

[0041] S3, after binarizing and sparse processing the first adjacency feature matrix in sequence, aggregate the neighbor node features to obtain aggregated features, perform parallel convolution processing and addition fusion processing on the aggregated features through several different graph convolutions, and then perform first feature compression processing, residual connection processing with the first brain region node feature matrix, and second feature compression processing in sequence to obtain a second brain region node feature matrix; After performing graph pooling processing on the first adjacency feature matrix, several edge convolution processing is performed in sequence to obtain a feature enhanced adjacency matrix, and after residual connection processing on the feature enhanced adjacency matrix and the first adjacency feature matrix, row convolution processing and column convolution processing are performed in sequence to obtain a second adjacency feature matrix; The node feature matrix and the second adjacency feature matrix of the second brain region were flattened and dimensionally reduced, and then spliced ​​to obtain the brain imaging omics feature map.

[0042] In this embodiment, the imaging genomics feature information is divided into 25 features of each brain region and the connection information between 90 brain regions, that is, a 90×25 first brain region node feature matrix and a 90×90 first adjacent feature matrix. Different extraction methods are designed for these two imaging genomics features to extract brain information in an all-round way. Specifically, the extraction of brain region node features is to regard 90 brain regions as 90 nodes, and each node contains 25 feature information (such as texture, intensity, etc.), thereby forming a 90×25 brain region node feature matrix as the first brain region node feature matrix. The first adjacent feature matrix is ​​binarized, and by setting a threshold (threshold = 0.8), the elements in the adjacency matrix that are greater than the threshold are set to 1, and the elements that are less than or equal to the threshold are set to 0, thereby obtaining the binarized adjacency matrix. .

[0043] In order to facilitate the subsequent graph neural network processing, it is necessary to convert the binary adjacency matrix into a sparse representation, that is, convert the adjacency matrix into an edge list. This sparse operation can be achieved through the dense_to_sparse function. The specific operation is as follows: , , Among them, threshold=0.8 is the set threshold, is the element in the i-th row and j-th column of the adjacency matrix, is the binary adjacency matrix element; is the edge list of the sparse representation, Represents a slave node To Node An edge of .

[0044] After sparse processing, the adjacency matrix is ​​converted from a dense matrix form to a sparse edge list representation, which is convenient for subsequent graph convolution operations. According to the edge list, all neighbor nodes that are connected to the current node are determined, and the features of these neighbor nodes are aggregated.

[0045] In this embodiment, a parallel graph convolution method is used to adopt a variety of aggregation strategies to extract the diverse features of each brain region, such as using the average, maximum, and minimum values ​​to extract the diverse features of each brain region, corresponding to three parallel graph convolutions. The three parallel graph convolution operations can capture the dependency relationship between each brain region and different neighboring nodes respectively, and then add and fuse these aggregated features, so that the node features fuse multiple levels of information. After the addition and fusion process, the dimension of the node features will increase.

[0046] In order to reduce the feature dimension, the node features are compressed by the first feature compression process (such as taking the average value), and the previous "multi-head features" are fused to obtain a more concentrated node feature representation. Assuming there is a node and its neighbor node set, the process of updating the node feature representation is as follows: , in, Representation Node In the The feature representation of the layer, is a trainable weight matrix, is the activation function RELU, It is an aggregate function, which can be maximum, minimum, average, etc.; Neighbor node In the The characteristics of the layer, Neighbor node In the The characteristics of the layer, Indicates that it belongs to a node v The set of neighbor nodes Nodes in u .

[0047] Furthermore, in order to prevent the gradient vanishing problem in deep networks and improve the efficiency of information flow, a residual connection mechanism is introduced to perform residual connection on the aggregated node features and the node feature matrix of the first brain region, retaining the original feature information and enhancing learning ability. Finally, the node features are subjected to second feature compression processing through the fully connected layer, further compressing the feature dimension to obtain the node feature matrix of the second brain region, which will represent the characteristics of each brain region in the entire network.

[0048] The extraction of the features of the first adjacency feature matrix is ​​as follows: for the 90×90 first adjacency feature matrix, it is first subjected to graph pooling processing, 50% of the nodes with more important connections are screened out, unimportant connections are removed, and the matrix structure is simplified to obtain a The matrix R is the real number field, that is, the elements of the matrix belong to the real number set.

[0049] The first adjacency feature matrix after graph pooling is subjected to several edge convolutions in sequence. Specifically, the edge convolutions are as follows: the first adjacency feature matrix after graph pooling is subjected to two-dimensional convolution using a convolution kernel of size 1× the number of nodes to obtain a feature matrix that is compressed and resized in a certain dimension, and then expanded in the spatial dimension to obtain an expanded feature matrix, and the expanded feature matrix is ​​transposed to obtain a transposed feature matrix, and element-by-element addition is performed based on the transposed feature matrix and the expanded feature matrix to obtain a correlation feature matrix. By introducing a transposition operation and adding the transposed matrix to the non-transposed matrix, it is ensured that information can be transmitted bidirectionally between all nodes of the graph, so that each node can receive forward and reverse information from other nodes.

[0050] Then, the correlation feature matrix is ​​globally pooled in both row and column directions. The row-direction pooling aggregates the relationship information between each brain region and other brain regions, revealing the overall output correlation of the brain region; the column-direction pooling aggregates the correlation information received by each brain region from other brain regions, showing the characteristics of the brain region as a receiver. The combination of row and column-direction pooling can extract the most valuable feature representation between brain regions on the basis of bidirectional information flow, which not only simplifies the feature dimension but also enhances the feature expression effect. After row global pooling and column global pooling, the row feature vector and column feature vector are obtained respectively. The row feature vector and column feature vector are multiplied and weighted with the first adjacency feature matrix to obtain the feature enhanced adjacency matrix. Not only the expression ability of node features is enhanced, but also the weights between nodes are dynamically adjusted, information redundancy is reduced, computational efficiency is improved, and the diversity of feature aggregation is increased. The specific operations are as follows: , , in, is the correlation feature matrix, is the feature-enhanced adjacency matrix, For the matrix A Perform a 3×3 two-dimensional convolution, is the transpose operation, is the global average pooling operation, and Operate in row and column directions respectively. is the element-wise dot product.

[0051] In this embodiment, the edge convolution process is cycled three times to gradually enhance the expressive power of feature information. After the above three edge convolution processes, a feature-enhanced adjacency matrix is ​​obtained. After the residual connection process of the feature-enhanced adjacency matrix, a row convolution and a column convolution operation are introduced, and row convolution and column convolution are performed in sequence. The row convolution process is a two-dimensional convolution using 1× the number of nodes to aggregate the information of all edges connected to a node to the node; the column convolution process is a two-dimensional convolution using the number of nodes × 1, and the node information aggregated by the row convolution process is aggregated to the node again, and finally the second adjacency feature matrix is ​​obtained.

[0052] It should be further explained that in order to preserve the original information, a 1×1 convolution is introduced for residual connection after the graph pooling process is performed on the first adjacency feature matrix of 90×90. At the end of the edge convolution process, the residual connection operation is performed on the obtained feature enhancement adjacency matrix and the first adjacency feature matrix after the 1×1 convolution process is residually connected to preserve the original information as much as possible.

[0053] Finally, the 90×25 second brain region node feature matrix and the 90×90 second adjacency feature matrix were flattened and input into a fully connected layer for dimensionality reduction. Then, the two feature matrices were concatenated, where the node features reflected the imaging genomics information of each brain region, and the adjacency matrix features reflected the connection relationship between brain regions. The combination of the two can provide more comprehensive brain information.

[0054] S4. After the brain image feature map is processed by convolution compression and flattening, it is spliced ​​with the brain imaging genomics feature map in the channel dimension to obtain preliminary fusion features. The preliminary fusion features and the brain imaging genomics feature map are linearly transformed respectively, and then processed by the attention mechanism to obtain weighted fusion features. Classification is performed based on the weighted fusion features to obtain brain age prediction results.

[0055] Specifically, the extracted brain image feature map It contains three-dimensional spatial information (H×W×D), that is, the features of each voxel point. In order to adapt to cross-modal feature fusion, it is necessary to first compress the brain image feature map by convolution operation, so that the channel dimension of the brain image feature map is adjusted to the same dimension as the brain imaging feature map, ensuring that they can be spliced ​​in the same dimensional space. Then, the three-dimensional spatial information of the image is flattened into a one-dimensional sequence through flattening operation to generate compressed brain image features. For the brain imaging feature map In order to effectively fuse with the compressed brain image features, it is compressed through the fully connected layer, and then the compressed brain imaging feature map is spliced ​​with the imaging feature in the channel dimension to generate the initial fusion feature , the formula is as follows: , in, , is the batch size, which indicates the number of data samples input into the model at one time. is the feature dimension of the image after convolution compression processing, is the compression dimension of radiomics features; is the brain image feature map, is a 3×3×3 three-dimensional convolution, For the flattening operation; is the brain imaging feature map, This concatenation operation retains the information of two types of features and ensures that features of different modalities are considered simultaneously.

[0056] In order to further capture the complex interactions between image features and radiomics features, a cross-modal attention mechanism is introduced. Specifically, the concatenated fusion features are transformed into Mapped to query vector Q, the radiomics features are mapped to key vector K and value vector V through linear transformation, and then processed using the scaled dot-product attention mechanism to perform dot product on the query vector Q and the key vector K to obtain the attention weight: , , in, To initially integrate features, Brain imaging feature map It is a linear transformation used to reduce the dimension; is the dimension of the key vector, The attention mechanism dynamically weights the radiomics features according to the correlation between the features of different modalities, so that the fused features can effectively capture the interactive information between the image and radiomics features.

[0057] After linear transformation of the preliminary fusion features and brain imaging feature maps, the attention mechanism is used to obtain weighted fusion features. Based on the weighted fusion features, they are compressed and flattened to obtain flattened fusion features. The flattened fusion features are classified using full connection to obtain brain age prediction results. , the formula is as follows: , in, is the activation function RELU, and are the weights and biases of the fully connected layer, is the flattening operation, is the fully connected layer compression, is the weighted fusion feature, It is a gender characteristic. is the bias of the fully connected layer.

[0058] In order to further improve the prediction accuracy, in another embodiment, the gender label is introduced as auxiliary information, and the gender label is processed by one-hot encoding to obtain the encoded gender feature ( ). The weighted fusion features are compressed and flattened to obtain the flattened fusion features. The flattened fusion features and gender features are spliced ​​and classified to obtain the brain age prediction results, which comprehensively considers a variety of information including images, imaging features and gender labels; the gender information and fusion features are spliced ​​and fully connected to perform regression prediction of brain age. The formula is as follows: , in, is the activation function RELU, and are the weights and biases of the fully connected layer, is the flattening operation, is the fully connected layer compression, is the weighted fusion feature, It is a gender characteristic. is the bias of the fully connected layer.

[0059] Image features, radiomics features, and gender labels are effectively integrated and ultimately used to predict brain age, improving the overall accuracy and expressiveness of brain age prediction.

[0060] The present invention also provides a brain age prediction system based on multimodal information fusion, comprising: Feature generation module: used to preprocess the acquired T1-weighted imaging to obtain a preprocessed image; after gray matter volume segmentation and brain region division, the imaging features of each brain region are obtained, and the first brain region node feature matrix is ​​generated accordingly; the imaging features of each brain region are weighted summed to obtain the feature value of each brain region; and the first adjacency feature matrix is ​​constructed based on the correlation between the feature values; Brain image feature extraction module: used for subjecting the preprocessed image to multiple multi-scale enhancement processing in sequence to obtain multiple feature enhancement maps, and performing splicing processing on the multiple feature enhancement maps to obtain a brain image feature map; the multi-scale enhancement processing is: subjecting the preprocessed image to multiple different convolution paths for parallel convolution processing and addition fusion processing to obtain a multi-scale feature map; performing channel feature enhancement processing and spatial feature enhancement processing on the multi-scale feature map in sequence to obtain a feature enhancement map, and using the feature enhancement map of this time as the preprocessed image in the next multi-scale enhancement processing; the number of channels of the multiple multi-scale enhancement processing increases with the increase of the number of processing times; Brain imaging omics feature extraction module: used to perform binarization and sparse processing on the first adjacency feature matrix in sequence, aggregate neighbor node features to obtain aggregated features, perform parallel convolution processing and addition fusion processing on the aggregated features through several different graph convolutions, and then perform first feature compression processing, residual connection processing with the first brain region node feature matrix, and second feature compression processing in sequence to obtain the second brain region node feature matrix; perform graph pooling processing on the first adjacency feature matrix, and then perform several edge convolution processing in sequence to obtain a feature enhanced adjacency matrix, perform row convolution processing and column convolution processing on the feature enhanced adjacency matrix and the first adjacency feature matrix after residual connection processing to obtain the second adjacency feature matrix; flatten and reduce the dimension of the second brain region node feature matrix and the second adjacency feature matrix, and then splice them to obtain a brain imaging omics feature map; Cross-modal feature fusion module: It is used to concatenate the brain image feature map with the brain imaging genomics feature map in the channel dimension after convolution compression and flattening to obtain preliminary fusion features. After linear transformation of the preliminary fusion features and the brain imaging genomics feature map, attention mechanism processing is performed to obtain weighted fusion features. Classification is performed based on the weighted fusion features to obtain brain age prediction results.

[0061] In addition, the present invention also provides a brain age prediction device based on multimodal information fusion, comprising a processor and a memory, wherein the processor implements the brain age prediction method based on multimodal information fusion as described above when executing a computer program stored in the memory.

[0062] Finally, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the brain age prediction method based on multimodal information fusion as described above.

[0063] Experimental content Experimental settings: The dataset was randomly divided into training set, validation set and test set according to the ratio of 75%, 15% and 10%. The experimental development environment used was pytorch1.9.0. On the NVIDIA RTX 2080 graphics processor, a total of 100 rounds were carried out during the training period. The Adam optimizer was used, the initial learning rate was 1e-3, and the batch size was set to 2. The brain age prediction method based on multimodal information fusion in this application was trained on the NVIDIA RTX 2080 graphics processor for about 6 hours. The experimental environment and specific settings are shown in Table 1, and the brain age prediction results are shown in Table 2.

[0064] Table 1

[0065] Table 2

[0066] The present invention provides a method, system, device and medium for predicting brain age based on multimodal information fusion. The method designs different feature extraction methods for different forms of input information. For image information, a multi-scale, multi-dimensional feature extraction and enhancement method based on dense connection is designed, which can capture multi-scale features and avoid the limitations of traditional convolutional neural networks (CNNs); for radiomics information, a parallel graph convolution processing method and a unique edge convolution and row-column convolution combination processing are designed, which are suitable for extracting complex graph structure information in non-Euclidean space, making up for the shortcomings of traditional Euclidean space methods in graph structure information extraction. The present invention performs multimodal feature fusion through feature splicing and cross-modal attention mechanism, and finally realizes accurate prediction of brain age. The present invention proposes a strategy that combines medical image data focusing on spatial information with radiomics features focusing on quantitative features and statistical information, fuses multimodal data such as structural information in images and quantitative information in radiomics, and applies it to the field of brain age prediction. Image information and imaging genomics information complement each other, exploring the relationship between brain structure information and brain age more comprehensively from multiple angles, laying the foundation for subsequent research on brain structure changes.

Claims

1. A brain age prediction method based on multimodal information fusion, characterized in that: The following steps are involved: S1, preprocessing the acquired T1-weighted imaging to obtain a preprocessed image; After gray matter volume segmentation and brain region division, the imaging genomics features of each brain region are obtained, and a first brain region node feature matrix is ​​generated accordingly. The imaging genomics features of each brain region are weighted summed to obtain the eigenvalue of each brain region, and a first adjacency feature matrix is ​​constructed based on the correlation between the eigenvalues; S2, subjecting the preprocessed image to multiple multi-scale enhancement processes in sequence to obtain multiple feature enhancement maps, and performing splicing processing on the multiple feature enhancement maps to obtain a brain image feature map; The multi-scale enhancement processing is as follows: the pre-processed image is subjected to parallel convolution processing and addition fusion processing through several different convolution paths respectively to obtain a multi-scale feature map; the multi-scale feature map is subjected to channel feature enhancement processing and spatial feature enhancement processing in sequence to obtain a feature enhancement map, and the feature enhancement map of this time is used as the pre-processed image in the next multi-scale enhancement processing; the number of channels of the multi-scale enhancement processing increases with the increase of the number of processing times; S3, after binarizing and sparse processing the first adjacency feature matrix in sequence, aggregate the neighbor node features to obtain aggregated features, perform parallel convolution processing and addition fusion processing on the aggregated features through several different graph convolutions, and then perform first feature compression processing, residual connection processing with the first brain region node feature matrix, and second feature compression processing in sequence to obtain a second brain region node feature matrix; After performing graph pooling processing on the first adjacency feature matrix, several edge convolution processing is performed in sequence to obtain a feature enhanced adjacency matrix, and after residual connection processing on the feature enhanced adjacency matrix and the first adjacency feature matrix, row convolution processing and column convolution processing are performed in sequence to obtain a second adjacency feature matrix; The node feature matrix of the second brain region and the second adjacent feature matrix were flattened and dimensionally reduced, and then spliced ​​to obtain the brain imaging omics feature map; S4. After the brain image feature map is processed by convolution compression and flattening, it is spliced ​​with the brain imaging genomics feature map in the channel dimension to obtain preliminary fusion features. The preliminary fusion features and the brain imaging genomics feature map are linearly transformed respectively, and then processed by the attention mechanism to obtain weighted fusion features. Classification is performed based on the weighted fusion features to obtain brain age prediction results.

2. The brain age prediction method based on multimodal information fusion according to claim 1, characterized in that: The method also includes obtaining gender labels, encoding them, and obtaining gender features; compressing and flattening the weighted fusion features to obtain flattened fusion features, and classifying the flattened fusion features and gender features after splicing to obtain brain age prediction results.

3. The brain age prediction method based on multimodal information fusion according to claim 1, characterized in that: The several different convolution paths in S2 are a single convolution process, or a combination of several convolution processes, or a combination of pooling process and convolution process; The preprocessed image is subjected to parallel convolution processing through several different convolution paths. The specific operation is as follows: the preprocessed image is subjected to parallel convolution processing through 4 different convolution paths. The first convolution path is a single 1×1×1 convolution processing, the second convolution path is a combination of 1×1×1 convolution processing and 3×3×3 convolution processing, the third convolution path is a combination of 1×1×1 convolution processing and 5×5×5 convolution processing, and the fourth convolution path is a combination of 3×3 maximum pooling processing and 1×1×1 convolution processing.

4. The brain age prediction method based on multimodal information fusion according to claim 1, characterized in that: In S3, after the first adjacency feature matrix is ​​subjected to graph pooling processing, several edge convolution processing is performed in sequence to obtain a feature enhanced adjacency matrix. The edge convolution processing is specifically as follows: after the first adjacency feature matrix after graph pooling processing is subjected to two-dimensional convolution processing, it is expanded in the spatial dimension to obtain an expanded feature matrix, the expanded feature matrix is ​​transposed to obtain a transposed feature matrix, element-by-element addition processing is performed based on the transposed feature matrix and the expanded feature matrix to obtain a correlation feature matrix, global pooling processing is performed on the correlation feature matrix in both row and column directions to obtain row feature vectors and column feature vectors correspondingly, and weighted processing is performed based on multiplication of the row feature vectors and the column feature vectors with the first adjacency feature matrix to obtain a feature enhanced adjacency matrix.

5. The brain age prediction method based on multimodal information fusion according to claim 1, characterized in that: In S2, channel feature enhancement processing and spatial feature enhancement processing are performed on the multi-scale feature map in sequence. The specific operation of the channel feature enhancement processing is: global pooling processing is performed on the multi-scale feature map by channel to obtain a global channel feature vector, and the global channel feature vector is subjected to channel-by-channel adaptive weight allocation processing, and then multiplied with feature maps of different sizes of the corresponding channel to obtain a channel feature enhancement map.

6. The brain age prediction method based on multimodal information fusion according to claim 5, characterized in that: The specific operation of the spatial feature enhancement processing is: globally pooling the channel feature enhancement map in three directions of the spatial dimension to obtain the global spatial feature vector in each direction, performing convolution and activation function processing in sequence based on the global spatial feature vector to obtain three spatial attention weights, multiplying the spatial attention weights with the global spatial feature vector in the corresponding direction to obtain a brain image feature map.

7. The brain age prediction method based on multimodal information fusion according to claim 1, characterized in that: In the S3, after the residual connection processing of the feature enhanced adjacency matrix, row convolution processing and column convolution processing are performed in sequence, specifically: a two-dimensional convolution of 1× the number of nodes is used for the feature enhanced adjacency feature matrix to aggregate the information of all edges connected to a node to the node; the column convolution processing is a two-dimensional convolution using the number of nodes × 1, and the node information aggregated by the row convolution processing is aggregated to the node again.

8. A brain age prediction system based on multimodal information fusion, characterized in that: include: Feature generation module: used to preprocess the acquired T1-weighted imaging to obtain a preprocessed image; After gray matter volume segmentation and brain region division, the imaging genomics features of each brain region are obtained, and a first brain region node feature matrix is ​​generated accordingly. The imaging genomics features of each brain region are weighted summed to obtain the eigenvalue of each brain region, and a first adjacency feature matrix is ​​constructed based on the correlation between the eigenvalues; Brain image feature extraction module: used to sequentially subject the preprocessed image to multiple multi-scale enhancement processing to obtain multiple feature enhancement maps, and to splice the multiple feature enhancement maps to obtain a brain image feature map; the multiple multi-scale enhancement processing is: subjecting the preprocessed image to multiple different convolution paths for parallel convolution processing and addition fusion processing to obtain a multi-scale feature map; sequentially subjecting the multi-scale feature map to channel feature enhancement processing and spatial feature enhancement processing to obtain a feature enhancement map, and use the feature enhancement map of this time as the preprocessed image in the next multiple multi-scale enhancement processing; the number of channels of the multiple multi-scale enhancement processing increases with the increase of the number of processing times; Brain imaging genomics feature extraction module: used to perform binarization and sparse processing on the first adjacency feature matrix in sequence, aggregate neighbor node features to obtain aggregated features, perform parallel convolution processing and addition fusion processing on the aggregated features through several different graph convolutions, and then perform first feature compression processing, residual connection processing with the first brain region node feature matrix, and second feature compression processing in sequence to obtain the second brain region node feature matrix; perform graph pooling processing on the first adjacency feature matrix, and then perform several edge convolution processing in sequence to obtain a feature enhanced adjacency matrix, perform row convolution processing and column convolution processing on the feature enhanced adjacency matrix and the first adjacency feature matrix after residual connection processing to obtain the second adjacency feature matrix; The node feature matrix of the second brain region and the second adjacent feature matrix were flattened and dimensionally reduced, and then spliced ​​to obtain the brain imaging omics feature map; Cross-modal feature fusion module: It is used to concatenate the brain image feature map with the brain imaging genomics feature map in the channel dimension after convolution compression and flattening to obtain preliminary fusion features. After linear transformation of the preliminary fusion features and the brain imaging genomics feature map, attention mechanism processing is performed to obtain weighted fusion features. Classification is performed based on the weighted fusion features to obtain brain age prediction results.

9. A brain age prediction device based on multimodal information fusion, characterized in that: The method comprises a processor and a memory, wherein the processor implements the brain age prediction method based on multimodal information fusion as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the brain age prediction method based on multimodal information fusion as described in any one of claims 1 to 7 is implemented.

Images