A method for identifying autism electroencephalogram by fusing multi-modal electroencephalogram information
Through the multimodal brain map information fusion recognition method, using the MRI sequence feature encoder, phenotypic feature embedding module and graph convolution module, the problems of strong subjectivity and insufficient model interpretability in autism diagnosis are solved, and higher diagnostic accuracy and sensitivity are achieved.
Patent Information
- Application Number
- CN202411947168.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-10-10
- Estimated Expiration
- 2044-12-27
AI Technical Summary
Existing technologies for autism diagnosis have problems such as highly subjective diagnostic results, failure to fully utilize multimodal magnetic resonance imaging data, and insufficient model interpretability.
A multimodal brain map information fusion and recognition method was adopted. Through the MRI sequence feature encoder, phenotypic feature embedding module, graph convolution module and MLP classifier, a training sample set was constructed and model training was performed. The structural MRI, functional MRI and non-image phenotypic data were integrated to extract brain region features and perform classification.
It improves the accuracy, sensitivity and heterogeneity of autism diagnosis, provides better performance indicators and achieves more objective diagnostic assistance.
Smart Images
Figure CN119888325B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image recognition, and in particular relates to a brain map recognition technology. Background Art
[0002] Autism Spectrum Disorder (ASD), also known as autism, is a common developmental disorder affecting infants, young children, and adolescents. Currently, autism diagnosis is primarily based on the diagnostician's judgment and autism-related scales, but the results are relatively subjective. Magnetic resonance imaging (MRI) provides objective biomarkers for autism diagnosis. Structural MRI (sMRI) and functional MRI (fMRI) provide information about the brain from both anatomy and functional connectivity perspectives. In recent years, deep learning has garnered increasing attention in the field of autism identification. While some methods have made initial progress, challenges persist, such as a failure to fully utilize multimodal MRI data and insufficient model interpretability. Summary of the Invention
[0003] To solve the above technical problems, the present invention proposes a method for identifying autism brain maps by fusing multimodal brain map information, which can obtain better performance indicators such as accuracy, sensitivity, and heterogeneity.
[0004] The technical solution adopted by the present invention is: a method for identifying autism brain maps by fusing multimodal brain map information, comprising:
[0005] S1. Preprocessing three modal data, namely, structural magnetic resonance imaging, functional magnetic resonance imaging, and non-image phenotypic data, of several labeled subjects to obtain corresponding brain region structural data, brain region functional data, and phenotypic data of each labeled subject. The brain region structural data, brain region functional data, and phenotypic data of each labeled subject are used as a sample to construct a training sample set; the labels include an autism label and a normal reference label;
[0006] S2. Construct a recognition model, which includes: an MRI sequence feature encoder, a phenotypic feature embedding module, a fusion module, a graph convolution module, and an MLP classifier;
[0007] The MRI sequence feature encoder generates a brain region connection edge feature matrix by fusing functional and structural magnetic resonance image information based on brain region structure data and brain region function data;
[0008] The phenotypic feature embedding module is used to extract phenotypic features from phenotypic data and generate a brain region node feature matrix by fusing two modal information: subject phenotype and brain region correlation;
[0009] The graph convolution module includes several layers of graph convolution networks connected in sequence, and an important node selection layer; the graph convolution module obtains significant brain regions of interest based on the input brain region connection edge feature matrix, brain region node feature matrix and corresponding sample labels;
[0010] The MLP classifier outputs a classification result based on the input significant brain regions of interest;
[0011] S3. Train the recognition model based on the training sample set constructed in step S1; use labeled sample data and loss definition function to iteratively train and adjust the learnable parameters of the recognition model to achieve the goal of convergence of loss value;
[0012] S4. Collect three modal data of the object to be identified, namely, structural magnetic resonance imaging, functional magnetic resonance imaging, and non-image phenotypic data, and preprocess them separately. Input the preprocessed brain region structural data, brain region functional data, and phenotypic data into the trained recognition model to obtain the classification result.
[0013] Beneficial effects of the present invention: The present invention proposes a method for identifying autism by fusion of multimodal brain map information, describing the brain map of each subject as BrainG(A,Z), where A represents the brain region connection edge feature and Z represents the brain region node feature. Brain region features are extracted from three modal data, namely structural magnetic resonance images, functional magnetic resonance images, and phenotypes, and then multimodal fusion is performed to construct a brain region map structure; a graph convolution module and loss function are designed to learn how to extract deep features of brain maps; based on the deep features of brain maps, a multi-layer perceptron (MLP) classifier is trained using labeled data from the ABIDE dataset to obtain better performance indicators such as accuracy, sensitivity, and heterogeneity; the trained multimodal brain map fusion model for identifying autism can be used to assist in the diagnosis and identification of autism. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 This is a block diagram of the multimodal brain map fusion recognition model;
[0015] Figure 2 embedding processes for phenotypic traits;
[0016] Figure 3 It is the graph convolution and TOP-K pooling layer;
[0017] Figure 4 It is the training process of multimodal brain map fusion recognition model;
[0018] Figure 5 Schematic diagram of important ROI-ROI connections related to autism. DETAILED DESCRIPTION
[0019] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.
[0020] like Figure 1 As shown, a multimodal brain map information fusion and recognition method of the present invention includes the following processing steps:
[0021] 1. Data Preprocessing
[0022] The ABIDE public dataset MRI imaging data (sMRIs structural magnetic resonance image sequences, fMRIs functional magnetic resonance image sequences) and non-imaging phenotypic data were preprocessed. Those skilled in the art should know that the ABIDE dataset includes 1009 subjects, including 493 ASD patients and 516 normal controls, and ASD and TC are the true labels corresponding to the patients.
[0023] 1.1: For MRI data, the existing medical segmentation atlas is used to divide the brain regions of interest (ROIs) of different modal MRI data. After processing, X f ∈R v×t 、X s ∈R v×3 Where v represents the number of ROIs divided according to the CC200 brain atlas. In this embodiment, v is 200. t represents the length of the blood oxygen concentration level dependent (BOLD) signal time series. In this embodiment, t is 100. 3 represents the three types of volume data extracted from sMRI, namely gray matter, white matter, and cerebrospinal fluid. R represents a real number set. f is the BOLD time series obtained after processing the original fMRI data, X s It is the volume data of gray matter, white matter, and cerebrospinal fluid obtained after preprocessing the original sMRI data.
[0024] 1.2: For phenotypic data, we consider representative information that most subjects have, including gender, age, full-scale IQ (FIQ), performance IQ (PIQ), verbal IQ (VIQ), handedness, and eye state at the time of the test. For gender, eye state at the time of the test, and handedness, we use a single hot encoding method. The remaining information, such as age, uses the original data. FIQ, PIQ, and VIQ are standardized to an average of 100 points. After normalization, the phenotypic data can be expressed as X p ∈R 7 .
[0025] 2. Generation of brain region edge connection features
[0026] Functional connectivity of brain regions is an important part of MRI analysis. It can reveal certain functional disorders or abnormalities and also plays an important role in the performance of subsequent classification and the mining of biomarkers.
[0027] This paper designs an MRI sequence feature encoder, which is implemented by a multi-head attention mechanism and can be used to obtain the brain region structure data X of the subject. s ∈R v×3 , brain region function data X f ∈R v×t Extract brain region structural features X′ s ∈R v×d′ , brain region functional characteristics X′ f ∈R v×d ; Where t represents the length of the time series, and in this embodiment, t is 100, v represents the number of ROIs, and in this embodiment, v is 200, and d′ represents the dimension of the feature, and here d′ is 8;
[0028] Then, the structural and functional brain region features are fused to obtain the MRI brain region fusion feature F MRI =X′ s +X′ f ,F MRI ∈R v×d ; Further construct the brain region connection edge feature matrix A = softmax(F MRI )[softmax(F MRI )] T ,A∈R v×v By extracting edge features from brain regions, we can fully utilize the complementary information of MRI multimodal data to obtain edge features A for graph classification, which reflects the dynamics and complex connection patterns of the data.
[0029] The specific process is described as follows:
[0030] 2.1: For the input T of the lth layer m (l-1) ∈R v×d′ , where: m∈{f,s} represents the functional and structural modal input data, v represents the number of ROIs (v=200), and d′ represents the feature dimension of the modal data. If it is fMRI modal data, d′=100; if it is sMRI modal data, d′=3. Convert to self-attention matrix Q m ,K m ,V m :
[0031]
[0032] in, is a learnable weight parameter,
[0033] 2.2: Calculate single-head attention according to formula (4):
[0034]
[0035] Where σ(·) is the softmax function.
[0036] The self-attention score can be used as a basis for potential biomarker identification. The value vector V m The larger the weight, the greater its contribution to the classification task.
[0037] 2.3: In order to achieve richer parallel features, a multi-head attention mechanism is introduced, and the i-th tap is defined as head i , whose expression is:
[0038]
[0039] in, Represents the weight matrix to be trained for the i-th tap.
[0040] 2.4: According to formula (6), the concatenation operation CONCAT is performed on the features of each tap, and then the concatenated features are combined with an output matrix W MLP Multiply them together to get the output of multi-layer self-attention.
[0041]
[0042] Among them, W MLP It is the learnable weight matrix used by the MRI sequence feature encoder to output the results, which is obtained by continuously updating through data sample training and learning. MLP ∈R hd×d , h is the number of attention taps, and in this embodiment, h is 2.
[0043] In this embodiment, the MRI sequence feature encoder includes a three-layer structure. The first layer input Represents X s or X f , the input of each subsequent layer is the output of the previous layer, and the output of the last layer is used as the output of the MRI sequence feature encoder.
[0044] 2.5: Multimodal MRI data is obtained after the encoder X′ f ∈R v×d and X′ s ∈R v×d Among them, X′ f and X′ s They represent fMRI features and sMRI features, respectively.
[0045] X′ f and X′ s is the output of the MRI sequence feature encoder, which uses a multi-head self-attention mechanism; it is equivalent to the following processing:
[0046] X′ f =Encoder(X f )
[0047] X′ s =Encoder(X s )
[0048] 2.6: According to formula (7), the fMRI and sMRI features are fused:
[0049] F MRI =X′ s +X′ f (7)
[0050] 2.7: Using fusion feature F MRI , generate brain region edge connection feature matrix:
[0051] A=softmax(F MRI )[softmax(F MRI )] T (8)
[0052] Where A∈R v×v Represents the generated brain region edge connection feature matrix, and softmax() represents the activation function.
[0053] 3. Brain region node feature generation
[0054] In order to find out the important phenotypic information features related to ASD classification, the present invention designed a phenotypic feature embedding module, such as Figure 2 As shown, Xp ∈R 7 For the phenotypic data of each subject, the phenotypic feature embedding module consists of three fully connected layers. Except for the last fully connected layer, the other two fully connected layers are followed by a nonlinear ReLU activation function layer and a Dropout layer to avoid overfitting. Figure 2 ReLu stands for linear rectification function, and its full spelling is Rectified Linear Unit; Dropout stands for random dropout, which is a professional term in this field.
[0055] First, the phenotypic data is processed by the embedding module to obtain the phenotypic characteristics, which can be expressed as
[0056] X′ p =f MLP (X p ) (9)
[0057] Among them, X′ p ∈R v×v .
[0058] Then, the brain region function data X f Perform Pearson correlation calculation to obtain the brain region correlation feature matrix Z′∈R v×v ;
[0059] Then, in order to promote the interaction of subsequent information in the brain map, the graph convolution classifier is guided to learn more features related to heterogeneity, and the phenotypic feature X′ is converted into p The brain region node feature matrix Z is obtained by fusing it with the brain region correlation feature Z′, which can be expressed as
[0060] Z=Z′+X′ p (10)
[0061] Where Z∈R v×v .
[0062] 4. Graph Convolution and Top-k Pooling
[0063] Design as Figure 3 Then, using 1009 subjects in the ABIDE dataset (including 493 ASD patients and 516 normal controls) as label data, the previous steps were used to obtain the brain region connection edge feature matrix and brain region node feature matrix of different subjects, allowing the graph convolution module to perform supervised learning, and continuously adjusting and optimizing the graph convolution module neural network link weights during training.
[0064] During the specific training process, the 1009 subjects will be divided into multiple batches; each training session will use the subject data from one of the batches.
[0065] The brain regions are very similar to graphs, so this paper designs and uses graph convolutional networks to learn how to extract deep features of brain maps. A k-layer graph convolutional network algorithm is used, and the node representation of each layer is updated through message passing and structural reconstruction. k is the node feature updated at the kth layer, and the update strategy is
[0066] h k =ReLu(Ah k-1 W k ) (11)
[0067] Among them, A is the brain region edge feature matrix, W k is the learnable weight parameter of the GCN (Graph Convolution Network) network, h 0 =Z.
[0068] To avoid numerical instability, a BatchNorm1D operation is appended after each graph convolution, denoted as
[0069] h k =BatchNorm1D(ReLu(Ah k-1 W k )) (12)
[0070] In this paper, we design an important node selection layer of Top-k pooling based on self-attention mechanism and set it at the end of graph convolutional network, such as Figure 3 shown.
[0071] Figure 3 In the above figure, Conv represents convolution, BatchNorm represents batch normalization, Leaky Relu represents Rectified Linear Unit (ReLU), Top-K Pooling represents Top-K pooling, Node Feature represents node feature, Top-K Loss represents Top-K loss, Attention score represents attention score, Self-Attention represents self-attention, sigmoid represents S-type activation function, and Top-K reserved nodes represents Top-K reserved nodes.
[0072] Through graph convolution learning, the obtained features are flattened and input into the MLP classifier, and all node features may have become saturated after multiple message diffusions. By sorting the importance scores of all nodes, the top k nodes can be retained while deleting other nodes. Using this Top-k pooling processing method, the most indicative nodes (significant ROIs) can be automatically retained, and the important brain areas related to the onset of ASD can be counted based on the CC200 brain atlas. At the same time, since most of the nodes are deleted, the dimension can also be reduced to avoid the dimensionality curse problem. In this embodiment, the value of k is 10.
[0073] The node feature representation after graph convolutional network processing is Where v is the number of ROIs, d output is the dimension of the output feature of the last convolutional layer. In order to decide which nodes to retain, the self-attention mechanism is introduced to learn the importance of the nodes. Q 、W K are two weight matrices to be learned, and the expression for calculating the node importance score is
[0074]
[0075] Where s∈R v Represents the importance score of each brain region.
[0076] Obviously, a higher importance score indicates a higher probability that the relevant node will be retained.
[0077] To improve interpretability, define
[0078]
[0079] This step can use penalty conditions to adjust the ability to calculate importance scores.
[0080] By scoring the selected nodes as 1 and the deleted nodes as 0, the graph can be sparser and interpretable significant brain region nodes can be obtained.
[0081] 5. Multilayer Perceptron
[0082] Multi-Layer Perception (MLP) was used as the classifier for ASD (Autism Spectrum Disorder) and TC (Typical Controls) classification recognition.
[0083] 6. Loss Function
[0084] Design the loss function of the model:
[0085] L=Lcl +αL sim +βL intra +γL inter +ξL TPK (15)
[0086] Among them, α, β, γ, ξ are hyperparameters.
[0087] Binary cross entropy loss function L cl Defined as
[0088]
[0089] Among them, y m,c is the true label, is the model's predicted value. b represents the number of subjects in the current batch, and c represents the number of classes. Instances specifically refer to the top k nodes with the highest scores found during training.
[0090] Similarity loss function L sim It is used to measure the difference between the shared representations of various modalities, which is defined as
[0091]
[0092] in represents the Frobenius norm.
[0093] The present invention treats the subjects with the same label in the current batch as a group, and each batch will correspond to two groups, one for subjects with autism label and one for subjects with normal reference label. Since individuals in the same group have consistent patterns in functional connectivity, the intra-group loss is considered, which is defined as
[0094]
[0095] Among them, ‖·‖2 represents the 2-norm, c∈C is a class, and the class here is the result obtained by the classifier, and the set S c ={i|Y i,c =1} contains all samples of subjects classified as c, A i Represents the brain region connection edge feature matrix A, Y generated by the sample of the i-th subject i,c =1 indicates that the MLP classifier output is 1. In this embodiment, the classifier output of 1 indicates autism classification, and the classifier output of 0 indicates normal reference classification. c and is the mean and variance of the A matrix generated by the data corresponding to the c-th subject in a group of the current batch. Within the same group, this loss function minimizes the difference by reducing the variance of the A matrix, thereby further ensuring a consistent pattern.
[0096] Due to the great differences in brain functional networks of different people, such as brain volume, gender, etc., the inter-group loss is considered, which is defined as
[0097]
[0098] where, μ a and μ b represent the mean values of the brain map deep features corresponding to groups a and b in the current batch. Groups a and b correspond to the groups of subjects with autism and the groups of subjects with normal reference respectively.
[0099] In the Top-k pooling layer based on the self-attention mechanism, the selected nodes and the unselected nodes are expected to have obviously opposite scores. Ideally, the span is at most within [0, 1], and the scores of the selected nodes and the unselected nodes should be 1 and 0 respectively. Therefore, the loss function of the mth instance is defined as
[0100]
[0101] where, b is the total number of the top k nodes with the highest scores found by training, s m represents the numerical value of the ROI in the brain region node feature matrix, represents the normalized numerical value of s m , is the score arranged in descending order, and the numerical value range is limited to [0, 1]. Obviously, L TPK will constrain the top k items and the remaining items to be 1 and 0 respectively, so as to disperse the two components.
[0102] The training optimization goal is to solve the minimum value of the loss function L of formula (15); in this embodiment, the training stop condition is Epoch = 500, that is, the training is completed after 500 iterations, and then the training is ended.
[0103] During the training process, the parameters that need to be adjusted in the entire recognition model are the weight matrices of each layer, which specifically include:
[0104] 1) The weight matrices to be trained of multiple taps in the MRI sequence feature encoder The weight matrix W MLP of the fully connected layer;
[0105] 2) The weight matrices of three fully connected layers in the phenotype feature embedding module;
[0106] 3) The learnable weight parameters W k of the graph convolutional neural network;
[0107] 4) The two weight matrices W Q to be learned in the Top-k pooling operation.、W K ;
[0108] 5) The fully connected weight matrix of the Multi-Layer Perception (MLP).
[0109] The specific adjustment method is based on minimizing the loss function according to the gradient descent algorithm, that is, continuously adjusting the weight matrix to reduce the error, thereby optimizing the performance of the recognition model.
[0110] The technical effects of the present invention are described below with reference to specific data:
[0111] The hyperparameter settings used in the model and method proposed in this invention are shown in Table 1.
[0112] Table 1 Hyperparameter settings
[0113]
[0114] In Table 1, Epoch represents the complete process of running the model to complete one forward propagation and backward propagation on all data; Batch size represents the division of the entire sample data into multiple batches, and the number of samples in each batch is called Batchsize; Learning rate represents the learning rate, which determines the speed and direction of the model updating weight parameters during training; Dropout rate represents the probability of randomly discarding a part of the neuron output during the training phase; Fold represents the number of folds (number of subsets), where Fold = 5, indicating that the training set is divided into 5 subsets and a 5-fold cross-validation method is used during training.
[0115] The dataset was divided into 70% for training, 10% for validation, and 20% for testing. The proposed method was compared with six state-of-the-art methods, and the performance comparison results are shown in Table 2. These results demonstrate that the proposed method achieved the highest classification accuracy of 79.4%. While also taking into account inter-subject heterogeneity, it also achieved a better balance in terms of sensitivity (80.3%) and specificity (78.4%).
[0116] Table 2 Performance comparison of different methods
[0117]
[0118] In Table 2, Accuracy represents accuracy, Sensitivity represents sensitivity accuracy, Specificity represents specificity accuracy, and AUC (Area Under Curve) represents the area under the receiver operating characteristic curve.
[0119] The loss during the training process of the multimodal brain map fusion recognition model can reach a convergence state, such as Figure 4 shown. Figure 4 Train Loss is the training loss and Loss is the loss value.
[0120] The comparison of the effects of different modal inputs on the graph convolution classifier is shown in Table 3. The results show that multimodal input can achieve the highest classification accuracy.
[0121] Table 3 Comparison of model classification effects of different modal inputs
[0122]
[0123] like Figure 5 As shown, through experiments, the present invention obtained the 10 most important ROI-ROI connections in the classification, and found that brain regions related to the frontal lobe, temporal lobe, and hippocampus were significantly associated with autism. Figure 5 In the figure, Left Lateral Occipital Cortex indicates the left lateral occipital cortex, Right Frontal Pole indicates the right frontal pole, Right Insular Cortex indicates the right insular cortex, Left Temporal Pole indicates the left temporal pole, Right Postcentral Gyrus indicates the right postcentral gyrus, Right Superior Frontal Gyrus indicates the right superior frontal gyrus, Right Hippocampus indicates the right hippocampus, Left Middle Temporal Gyrus indicates the left middle temporal gyrus, Right Frontal Pole indicates the right frontal pole, Right Superior Temporal Gyrus indicates the right superior temporal gyrus, Right Frontal Pole indicates the right frontal pole, None indicates None, Right Supramarginal Gyrus indicates the right supramarginal gyrus, Left Temporal Pole indicates the left temporal pole, Right Superior Temporal Gyrus indicates the right superior temporal gyrus, Left Frontal Pole indicates the left frontal pole, Right Putamen indicates the right nucleus, Right Parahippocampal Gyrus indicates the right parahippocampal gyrus, Left ParahippocampalGyrus indicates the left parahippocampal gyrus.
[0124] Those skilled in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the principles of the present invention, and it should be understood that the scope of the present invention is not limited to such specific descriptions and embodiments. Various modifications and variations are readily apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims.
Claims
1. A method for identifying autism brain maps by fusion of multimodal brain map information, characterized in that: include: S1. Preprocess the structural MRI, functional MRI, and non-image phenotypic data of several labeled subjects separately to obtain the corresponding brain region structural data, brain region functional data, and phenotypic data of each labeled subject. The brain region structural data, brain region functional data, and phenotypic data of each labeled subject are taken as a sample to construct a training sample set. The labels include autism labels and normal reference labels; S2. Construct a recognition model, which includes: an MRI sequence feature encoder, a phenotypic feature embedding module, a fusion module, a graph convolution module, and an MLP classifier; The MRI sequence feature encoder generates a brain region connection edge feature matrix by fusing functional and structural magnetic resonance image information based on brain region structure data and brain region function data; The phenotypic feature embedding module is used to extract phenotypic features from phenotypic data and generate a brain region node feature matrix by fusing two modal information: subject phenotype and brain region correlation; The graph convolution module includes several layers of graph convolution networks connected in sequence, and an important node selection layer; the graph convolution module obtains significant brain regions of interest based on the input brain region connection edge feature matrix, brain region node feature matrix and corresponding sample labels; The MLP classifier outputs a classification result based on the input significant brain regions of interest; S3. Train the recognition model based on the training sample set constructed in step S1; use labeled sample data and loss definition function to iteratively train and adjust the learnable parameters of the recognition model to achieve the goal of convergence of loss value; S4. Collect three modal data of the object to be identified, namely, structural magnetic resonance imaging, functional magnetic resonance imaging, and non-image phenotypic data, and preprocess them separately. Input the preprocessed brain region structural data, brain region functional data, and phenotypic data into the trained recognition model to obtain the classification result.
2. The method for identifying autism brain maps by fusion of multimodal brain map information according to claim 1, characterized in that: The MRI sequence feature encoder is implemented using a multi-head attention mechanism.
3. The method for identifying autism brain maps by fusion of multimodal brain map information according to claim 2, characterized in that: The process of constructing the brain region connection edge feature matrix is as follows: The brain region structure data of the input MRI sequence feature encoder is recorded as X s , brain region function data is X f ; The brain region structural features output by the MRI sequence feature encoder are recorded as X′ s , the functional characteristics of the brain region are recorded as X′ f ; X′ f =Encoder(X f ) X′ s =Encoder(X s ) in, It is used to represent the input of the first layer of the MRI sequence feature encoder. The input of the first layer of the MRI sequence feature encoder is represented as Refers to X s or X f , CONCAT represents the concatenation operation; head1,head2,…,head h Represents each tap in the multi-head attention mechanism; W MLP Represents the learnable weight matrix used by the MRI sequence feature encoder to output the results; head i represents the i-th tap, Represents the weight matrix to be trained for the i-th tap; For X′ s and X′ f Fusion is performed to obtain the fusion feature F MRI ; Using fusion feature F MRI , generate brain region edge connection feature matrix: A=softmax(F MRI )[softmax(F MRI )] T Among them, softmax() represents the activation function and the superscript T represents transposition.
4. The method for identifying autism brain maps by fusion of multimodal brain map information according to claim 3, characterized in that: The phenotypic feature embedding module consists of three fully connected layers. Except for the last fully connected layer, the other two fully connected layers are followed by a nonlinear ReLU activation function layer and a Dropout layer.
5. The method for identifying autism brain maps by fusion of multimodal brain map information according to claim 4, characterized in that: The process of extracting phenotypic features is: Input non-image phenotypic data X p , after being processed by the phenotypic feature embedding module, the phenotypic feature X′ is obtained p ; Brain area function data X f Perform Pearson correlation calculation to obtain the brain region correlation feature matrix Z′; Fusion phenotypic feature X′ p And the brain region correlation feature matrix Z′, the brain region node feature matrix Z is obtained.
6. The method for identifying autism brain maps by fusion of multimodal brain map information according to claim 5, characterized in that: The important node selection layer is implemented based on Top-k pooling of the self-attention mechanism.
7. The method for identifying autism brain maps by fusion of multimodal brain map information according to claim 6, characterized in that: The loss function during training is: L=L cl +αL sim +βL intra +γL inter +ξL TPK Among them, α, β, γ, ξ are hyperparameters; y m,c is the true label of the mth subject of category c, is the predicted value of the recognition model for the mth subject of category c, b represents the number of subjects in the current training batch, and c represents a category output by the MLP classifier; represents the Frobenius norm; ||·||2 represents the 2-norm, c∈C, C includes the different classifications output by the MLP classifier, and the set S c Contains samples of all subjects with label c, A i represents the brain region connection edge feature matrix A generated by the sample of the i-th subject, μ c and is the mean and variance of the brain region connection edge feature matrix A generated by the samples corresponding to the subjects in the current batch group; μ a represents the mean of the brain regions of interest corresponding to the group of subjects whose true label is autism in the current batch, μ b Indicates the mean of the brain regions of interest corresponding to the group of subjects whose true labels in the current batch are normal references; It represents the attention score of the i-th brain region of interest corresponding to the m-th subject selected in the current batch, It represents the attention score of the i+kth brain region of interest corresponding to the mth subject who was not selected in the current batch, and it represents the total number of brain regions of interest for each subject.