A multimodal autism diagnostic method, system, medium and product based on resting-state fMRI and phenotypic text information

Abstract

This invention discloses a multimodal autism diagnosis method, system, medium, and product based on resting-state fMRI and phenotypic text information. This method processes fMRI and phenotypic text information using a pre-trained heterogeneous graph neural network model to output diagnostic results. The method uses resting-state fMRI to construct a brain functional connectivity map, encoding the phenotypic text information into a two-dimensional vector. The method then extracts brain connectivity feature embeddings from the fMRI modality and phenotypic information feature embeddings from the text modality. These feature embeddings are fused using a gated fusion network. A heterogeneous population graph is constructed based on the fused features and the similarity of the phenotypic information. Dual-channel information aggregation and adaptive feature fusion are then used on the heterogeneous population graph to generate population graph features for outputting diagnostic results. This invention aims to improve the accuracy of autism diagnosis using multimodal information and is applicable to fields such as neuroimaging analysis and intelligent healthcare.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and specifically relates to a multimodal autism diagnosis method, system, medium and product based on resting-state fMRI and phenotypic text information. Background Art

[0002] Autism spectrum disorder is a serious and widespread public health problem worldwide. Autistic patients experience abnormalities in cognitive, emotional, and behavioral functions. Due to the lack of clear pathological mechanisms and reliable biomarkers as a reference, doctors' diagnostic results are easily affected by personal subjective judgment.

[0003] The brain is usually viewed as a complex network system, and functional magnetic resonance imaging technology has become an important tool for studying brain network functionality in recent years. Brain functional connectivity analysis based on resting-state functional magnetic resonance imaging (rs-fMRI) has been widely used in the study of brain network disorders. These studies provide important basis for understanding the neural mechanisms of autism spectrum disorder by extracting and analyzing the functional connectivity characteristics between brain regions.

[0004] In recent years, graph neural networks have provided a new paradigm for modeling brain functional connectivity networks with their topological feature extraction capabilities and relational reasoning advantages. Existing research mainly uses two functional brain network modeling strategies: the individual graph method defines functional brain areas as nodes and functional connection strengths as edges; the group graph method defines subjects as nodes and constructs edges by calculating the similarity of the subjects' phenotypic text information.

[0005] Existing methods face multiple bottlenecks. In the individual graph method, its nodes represent specific functional brain areas, and graph convolution is used to extract the brain functional connectivity features of a single subject, which can locate disease-related abnormal brain areas, but there are deficiencies in predictive accuracy. The group graph method uses phenotypic text information to calculate the similarity between subjects to construct a relationship network, and extracts the relationship features between groups through graph convolution, which has a high accuracy rate, but its nodes only represent abstract individual characteristics, making it difficult to locate specific functional brain areas, which weakens the identification of biomarkers. Only using the brain functional connectivity features of a single modality for diagnosis ignores the fact that the phenotypic text information also contains some features that are very important for diagnosis, and cannot fully utilize the comprehensive characteristics of the subject. In addition, during the acquisition of resting-state functional MRI data, due to significant differences in instrumentation, data acquisition protocols, subject population characteristics, and preprocessing methods among different acquisition sites, the data show obvious heterogeneity in dimensions such as quality, resolution, and signal-to-noise ratio. The existence of this data heterogeneity will have a certain impact on diagnosis and easily lead to subjective misjudgment. Summary of the Invention

[0006] The purpose of the present invention is to provide a multimodal autism diagnosis method, system, medium and product based on resting-state fMRI and phenotypic text information to solve the problems existing in the above-mentioned prior art and reduce the decision-making risk of autism diagnosis.

[0007] In order to achieve the above object, the present invention provides the following solutions:

[0008] The present invention provides a multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information, comprising the following steps:

[0009] S1: Obtain the resting-state functional magnetic resonance imaging (fMRI) and phenotypic text information of the input individual, construct a brain functional connectivity map, and encode the phenotypic text information into a two-dimensional vector. ;

[0010] S2: Processing brain functional connectivity maps to obtain brain connectivity feature embedding , processing two-dimensional vectors Obtain phenotypic information feature embedding ;

[0011] S3: Fusion of brain connectivity feature embeddings via a gated fusion network and phenotypic information feature embedding , get the fusion features ;

[0012] S4: Based on fusion features Construct a heterogeneous population graph, extract population-level features and input them into the classifier to obtain the diagnosis results of autism;

[0013] Furthermore, in S1, the resting-state functional magnetic resonance imaging (fMRI) and phenotypic text information of the input individual are obtained to construct a brain functional connectivity map, and the phenotypic text information is encoded into a two-dimensional vector. , specifically as follows;

[0014] S1.1, constructing a brain functional connectivity map based on resting-state fMRI input;

[0015] Resting-state functional magnetic resonance imaging (rs-fMRI) data were preprocessed. The brain was divided into 200 regions using the CC200 brain atlas template. The average time series of each brain region was extracted and the average time series of the 200 brain regions based on blood oxygen level dependence was obtained. , where N is the number of brain regions divided and T is the length of the time series (in milliseconds). The Pearson correlation coefficient between the time series of each brain region and the time series of other brain regions is calculated to obtain a functional connectivity matrix To construct a brain functional connectivity map ,in is a collection of nodes, each node represents a brain region, is a set of edges containing the functional connectivity information between each brain region.

[0016] First, the functional connectivity matrix Fisher's r-to-z transformation was applied to make the Pearson correlation coefficient distribution closer to the normal distribution. Each brain region was regarded as a node in the graph, and the node features were directly adopted from the corresponding row vector in the transformed functional connectivity matrix to represent the functional connectivity relationship between the brain region and all other brain regions. For the construction of edges, the absolute value of the functional connectivity matrix was taken and converted into the initial fully connected graph. Then, the self-loop connections were removed and the threshold was applied for filtering. Only the edges with functional connectivity strength greater than the threshold were retained. Finally, the edge weights were directly adopted from the Pearson correlation coefficient between the brain regions.

[0017] S1.2, encode phenotypic text information into a two-dimensional vector ;

[0018] Specifically, the two categories of text information, site information and gender information, are first uniquely encoded; then the continuous text information is normalized, including normalizing age and three IQ indicators (verbal IQ VIQ, performance IQ PIQ and full IQ FIQ) by subtracting the mean and dividing the standard deviation. Finally, the preprocessed phenotypic text information data is encoded into a two-dimensional vector ,in represents the number of subjects, It is the feature dimension of the subject's phenotype text information.

[0019] Furthermore, in S2, the brain functional connectivity map is processed to obtain brain connectivity feature embedding , processing two-dimensional vectors Obtain phenotypic information feature embedding , specifically as follows;

[0020] S2.1. Extracting brain connectivity feature embedding from brain functional connectivity graph using graph convolutional network and contrastive mutual information feature selection ;

[0021] Extracting brain connectivity feature embedding from brain functional connectivity maps using graph convolutional networks and contrastive mutual information feature selection When the graph convolution network is a 2-layer GCN graph convolution network layer, the contrastive mutual information feature selection is a mutual information feature selection method based on contrastive learning, specifically as follows: given node feature , first use a GCN graph convolution layer with an output channel of 1 to estimate the importance score of each node ,Then, select the k nodes with the highest scores to retain. In the process of selecting these k important nodes, firstly, Sort all nodes, select the k nodes with the highest scores, and obtain their indexes. Then, apply the tanh activation function to the importance scores of the selected k nodes, map these score values to the range of (-1, 1), and use them as the weight coefficients of the nodes. By multiplying the index features and weight coefficients element by element, the node features after node selection can be obtained. At the same time, the connection structure of the graph is reconstructed through the filtering operation, only the edges between the selected nodes are retained, and the corresponding edge indexes and attributes are updated to form a new adjacency matrix , the specific operations are as follows:

[0022] ,

[0023] ,

[0024] ,

[0025] .

[0026] In the above formula and Represent the node feature matrix and adjacency matrix respectively, Represents a graph convolution operation with a layer output channel of 1, represents the importance score of the output node, The top-k operation is used to select the k node indexes with the highest scores. topk(.) represents the top-k operation, which is an operation that can select the most important k targets according to their importance. The feature matrix of the selected node is and the score after tanh activation Multiply element by element to get the new node feature matrix , Represents the new adjacency matrix formed.

[0027] In the process of contrastive mutual information feature selection, a contrastive mutual information maximization strategy based on structure perception is used. Mutual information is the core concept in information theory that quantifies the amount of information shared by two random variables. In contrastive mutual information feature selection, the focus is on the mutual information between the node importance score and the original node features. By maximizing this mutual information, it is ensured that the node importance score Y generated by graph convolution can accurately evaluate the importance of the original node. At the same time, the perception of the graph topology structure is enhanced through the contrastive learning method based on structure perception. InfoNCE is a contrastive learning method for estimating and maximizing mutual information. In contrastive mutual information feature selection, the node embedding representation Y is first calculated through graph convolution. Y is the node importance score. Then, a contrastive mutual information estimation loss based on structure perception is constructed based on InfoNCE. , this contrastive mutual information estimation loss can effectively quantify and maximize the mutual information between the node embedding Y and the original node features, which is defined as follows:

[0028] ,

[0029] in is the graph convolutional embedding representation of the node, is the original node feature, It is a negative sample feature constructed based on node degree, and MLP is a multi-layer perceptron used to evaluate the mutual information between feature pairs.

[0030] The positive sample pairs are constructed in the following way: for each node i, the embedding representation generated by graph convolution is The corresponding original features Pairing forms positive sample pairs.

[0031] The negative sample construction adopts a sampling strategy based on the topological heterogeneity of the brain network: first, the degree values of all nodes in the graph are calculated, and the nodes are divided into a high-degree node set and a low-degree node set with the average degree as the threshold. For high-degree nodes, random sampling is performed from the low-degree node set as negative samples; for low-degree nodes, random sampling is performed from the high-degree node set as negative samples. The embedding representation generated by graph convolution is Paired with the negative samples constructed based on node degree to form negative sample pairs.

[0032] The final output of the contrastive mutual information feature selection is generated by further applying the GCN graph convolution network layer to the selected k node features for residual graph convolution operation:

[0033] ,

[0034] in and Represents the new node feature matrix and adjacency matrix obtained after comparative mutual information feature selection, Represents the final output of contrastive mutual information feature selection, i.e. brain connection feature embedding .

[0035] S2.2, using the constructed multimodal feature extractor to process the two-dimensional vector Obtain phenotypic information feature embedding :

[0036] A multimodal feature extraction module is constructed to extract To extract more representative feature representations, the multimodal feature extraction module is implemented using a multi-layer perceptron (MLP) structure, which includes two hidden layers. Each layer combines linear transformation, batch normalization, ReLU activation function, and Dropout regularization technology to improve the generalization and expression capabilities of the model. The phenotypic feature extraction process can be expressed as follows:

[0037] ,

[0038] in, and Represent the original phenotypic features and the phenotypic features with more potential after extraction, and each hidden layer function Defined as:

[0039] ,

[0040] in, and They represent the trainable parameters of the i-th hidden layer respectively. BatchNorm represents the normalization layer, which helps to accelerate network convergence and improve training stability. The ReLU activation function enables the network to learn complex feature interactions and effectively alleviates the gradient vanishing problem in deep networks. The Dropout layer reduces the risk of the model over-relying on certain features by randomly inactivating neurons, and ultimately outputs more refined phenotypic information features of the phenotypic text information data. , that is, phenotypic information feature embedding , which captures the most relevant features for disease diagnosis in phenotypic text information data.

[0041] Furthermore, in S3, brain connectivity feature embedding is fused through a gated fusion network. and phenotypic information feature embedding , get the fusion features , specifically as follows;

[0042] Embedding brain connectivity features using gated fusion networks and phenotypic information feature embedding They are processed independently, mapped to the hidden space of the same dimension through their respective linear transformation layers, and the ReLU activation function is applied to enhance the nonlinear expression ability to obtain the transformed feature representation and , as shown below:

[0043] ,

[0044] ,

[0045] ,

[0046] in, and They represent the trainable parameters, and are the results of linear transformation and ReLU activation of brain functional connectivity features and phenotypic features, respectively. and Splicing is performed on the feature dimension to form a higher-dimensional joint feature representation H. Based on this spliced feature H, the network calculates a gate value gate through a gating unit containing a linear layer and a Sigmoid activation function. The Sigmoid function ensures that the gate value is between 0 and 1, indicating the fusion ratio of the two features:

[0047] ,

[0048] Among them, fused is the splicing feature H mentioned above. Finally, the network uses this adaptive gating value to perform weighted fusion of the features of the two channels to obtain the final fused feature representation:

[0049] ,

[0050] Among them, gate is the gate value, is the output fusion feature.

[0051] Furthermore, in S4, based on the fusion feature Construct a heterogeneous population graph, extract population-level features and input them into the classifier to obtain the diagnosis results of autism, as follows;

[0052] S4.1, construct heterogeneous group graph:

[0053] Heterogeneous group graph consists of node sets and edge sets Composition, expressed as , where the node set Each node in represents a subject, the total number of nodes is n, and the edge set Indicates the data collection site relationship between different subjects. According to the different collection sites of the subject data, the edges are divided into the same collection site edges. and different collection sites , the individual feature embedding obtained after each subject is processed by the individual graph model is used as the node feature of the group graph , where h is the feature dimension of the brain connectivity feature embedding and phenotypic information feature embedding fusion feature obtained by each subject after the individual graph model. The following phenotypic information data similarity score calculation method is used: for categorical variables, if the values of two individuals are the same, they are considered to be similar; for numerical variables, the similarity is judged based on whether the difference between the two values is less than the set threshold. Finally, this process generates a phenotypic similarity matrix , using the phenotypic information data similarity score as the weight of the edge between nodes, It is a threshold function that controls the sparsity of the group graph by retaining only the edges with weights greater than the set threshold, excluding weak associations, and finally obtaining the adjacency matrix Used to construct a group graph and, at the same time, construct a mask matrix based on site information , further divide the edges into similar edges (same site) and heterogeneous edges (different sites), and finally, combine the adjacency matrix and the mask matrix As well as the individual feature embedding extracted by the individual graph model, a heterogeneous group graph is constructed.

[0054] S4.2, dual-channel information aggregation and adaptive feature fusion:

[0055] The network consists of three layers, each of which consists of two parallel TransformerConv graph convolutions, which aggregate the information of neighbors of the same site and neighbors of different sites respectively, and fuse homogeneous features and heterogeneous features through an adaptive weight mechanism. The initial fusion weight of homogeneous features is set to 0.8, and the initial fusion weight of heterogeneous features is set to 0.2. In the subsequent information aggregation process, learnable feature fusion weights are used. The fusion weights are dynamically adjusted in an adaptive manner during model training to aggregate homogeneous and heterogeneous information. The dual-channel information aggregation and adaptive feature fusion process is shown below:

[0056] ,

[0057] ,

[0058] ,

[0059] in, , as well as They represent the node feature set, homogeneous edge set, and heterogeneous edge set of the input L layer respectively. GraphTransformer represents the information aggregation process of TransformerConv graph convolution. and denote the feature embeddings learned from the same-site and different-site neighbors, respectively. and is the trainable homogeneous and heterogeneous feature fusion weight of the Lth layer, and each layer is updated by a normalization operation so that its sum is 1, as shown below:

[0060] ,

[0061] ,

[0062] The output features of each layer are processed by Batchnorm normalization and LeakyReLU activation function, and then used as the input of the next dual-channel information aggregation layer and saved as And cascade with the output features of the subsequent layers to obtain group-level features :

[0063] ,

[0064] ,

[0065] in Represent the output of each layer respectively, and finally, the group level features Input a fully connected layer classifier to obtain the diagnosis result of autism.

[0066] In addition, the present invention also provides a multimodal autism diagnostic system based on resting-state fMRI and phenotypic text information, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the multimodal autism diagnostic method based on resting-state fMRI and phenotypic text information.

[0067] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multimodal autism diagnostic method based on resting-state fMRI and phenotypic text information through a processor.

[0068] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multimodal autism diagnostic method based on resting-state fMRI and phenotypic text information through a processor.

[0069] Compared with existing technologies, the present invention has the following major advantages: it combines the advantages of individual and group graphs to achieve higher diagnostic accuracy. It also utilizes a multimodal feature extraction module to extract phenotypic information feature embeddings from phenotypic text information that complement brain connectivity feature embeddings, thereby integrating feature embeddings from different modalities and fully utilizing multimodal data to effectively improve diagnostic accuracy. Furthermore, due to significant differences in instrumentation, data acquisition protocols, subject population characteristics, and preprocessing methods during the collection process of resting-state functional magnetic resonance imaging (fMRI) data, different acquisition sites exhibit significant heterogeneity in terms of data quality, resolution, and signal-to-noise ratio. To address this issue, a heterogeneous group graph is constructed based on the relationships between data acquisition sites. Dual-channel information aggregation and adaptive feature fusion methods are then used to extract group-level features for classification, effectively addressing the problem of data heterogeneity. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of the basic flow of the method according to an embodiment of the present invention;

[0071] Figure 2 An algorithm framework diagram of a multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information provided by an embodiment of the present invention;

[0072] Figure 3 To obtain brain connection feature embedding in the embodiment of the present invention and phenotypic information feature embedding Schematic diagram of working principle;

[0073] Figure 4 Schematic diagram of the working principle of the comparative mutual information feature selection method in an embodiment of the present invention;

[0074] Figure 5 Schematic diagram of the working principle of the gated fusion network model in an embodiment of the present invention;

[0075] Figure 6 Schematic diagram of the working principle of the dual-channel information aggregation and adaptive feature fusion network in an embodiment of the present invention;

[0076] Figure 7 This is a visualization diagram of the classification effect under the ABIDE I dataset in an embodiment of the present invention, where: Figure 7 (a) shows the distribution of original features in the ABIDE I dataset. Figure 7 (b) shows the distribution of sample features after classification;

[0077] Figure 8 This is a visualization diagram of the classification effect under the ABIDE II dataset in an embodiment of the present invention, where: Figure 8(a) shows the distribution of original features in the ABIDE II dataset. Figure 8 (b) shows the distribution of sample features after classification. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0079] The core of the present invention is to provide a multimodal autism diagnostic method, system, medium and product based on resting-state fMRI and phenotypic text information to solve the problems existing in existing methods and technologies.

[0080] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0081] Figure 1 A schematic diagram of the basic flow of a multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information provided by an embodiment of the present invention includes the following steps:

[0082] S1: Obtain the resting-state functional magnetic resonance imaging (fMRI) and phenotypic text information of the input individual, construct a brain functional connectivity map, and encode the phenotypic text information into a two-dimensional vector. ;

[0083] S2: Processing brain functional connectivity maps to obtain brain connectivity feature embedding , processing two-dimensional vectors Obtain phenotypic information feature embedding ;

[0084] S3: Fusion of brain connectivity feature embeddings via a gated fusion network and phenotypic information feature embedding , get the fusion features ;

[0085] S4: Based on fusion features A heterogeneous group graph is constructed, and group-level features are extracted and input into the classifier to obtain the diagnosis results of autism.

[0086] Figure 2The present invention provides an algorithm framework diagram of a multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information, which consists of two parts: an individual graph and a group graph. In the individual graph, a brain functional connection graph of each subject is constructed, and the individual brain connection feature embedding is extracted by two layers of GCN graph convolution, and the mutual information feature selection method based on contrastive learning is used to remove irrelevant features. In order to make full use of non-imaging phenotypic text information data, a multimodal feature extraction module is introduced to extract features from the phenotypic text information data, and the phenotypic information feature embedding and the individual brain connection feature embedding are fused using a gated fusion network as node features in the group graph. In the group graph, a heterogeneous group graph based on the relationship between data collection sites is constructed, with the subjects as nodes, and the node relationship is described using the same-site edge and the different-site edge. The edge weight is calculated by the similarity of the phenotypic information data. Finally, group-level features are extracted through dual-channel information aggregation and adaptive feature fusion methods and fed into the fully connected layer to distinguish healthy controls from autistic patients, i.e., HC and ASD.

[0087] like Figure 3 As shown, in step S1 of this embodiment, the resting state functional magnetic resonance imaging (FSFMRI) and phenotypic text information of the input individual are obtained, and the brain functional connectivity map is constructed using the resting state functional magnetic resonance imaging (FSFMRI). The phenotypic text information is encoded into a two-dimensional vector , specifically as follows;

[0088] S1.1, construct brain functional connectivity map based on input resting-state functional magnetic resonance imaging:

[0089] Resting-state functional magnetic resonance imaging (rs-fMRI) data were preprocessed. The brain was divided into 200 brain regions using the CC200 brain atlas template. The average time series of each brain region was extracted and the average time series of the 200 brain regions based on blood oxygen level dependence was obtained. , where N is the number of brain regions and T is the length of the time series (in milliseconds), the Pearson correlation coefficient between each brain region time series and the other brain regions time series was calculated to obtain a functional connectivity matrix To construct a brain functional connectivity map ,in is a collection of nodes, each node represents a brain region, is a set of edges containing the connection information between each brain region.

[0090] First, the functional connectivity matrix Fisher's r-to-z transformation was applied to make the Pearson correlation coefficient distribution closer to the normal distribution. Each brain region was treated as a node in the graph, and the node features were directly adopted from the corresponding row vector in the transformed functional connectivity matrix to represent the functional connectivity between the brain region and all other brain regions. For edge construction, the absolute value of the functional connectivity matrix was taken and converted into the initial fully connected graph. Self-loop connections were then removed and a threshold was applied for filtering. Only edges with functional connectivity strength greater than the threshold were retained. Finally, the edge weights were directly adopted from the Pearson correlation coefficients between brain regions.

[0091] S1.2, encode phenotypic text information into a two-dimensional vector :

[0092] Specifically, the two categories of text information, site information and gender information, are first uniquely encoded; then the continuous text information is normalized, including normalizing age and three IQ indicators (verbal IQ VIQ, performance IQ PIQ and full IQ FIQ) by subtracting the mean and dividing the standard deviation. Finally, the preprocessed phenotypic text information data is encoded into a two-dimensional vector ,in represents the number of subjects, It is the phenotypic characteristic dimension of the subject.

[0093] like Figure 3 As shown, in step S2 of this embodiment, the brain functional connectivity map is processed to obtain the brain connectivity feature embedding , processing two-dimensional vectors Obtain phenotypic information feature embedding , specifically as follows;

[0094] S2.1. Extracting brain connectivity feature embedding from brain functional connectivity graph using graph convolutional network and contrastive mutual information feature selection :

[0095] Extracting brain connectivity feature embedding from brain functional connectivity maps using graph convolutional networks and contrastive mutual information feature selection When , the graph convolutional network is a 2-layer GCN graph convolutional network layer, and the contrastive mutual information feature selection is a mutual information feature selection method based on contrastive learning, such as Figure 4 As shown, specifically as follows, given node features , first use a graph convolutional network (GCN) layer with an output channel of 1 to estimate the importance score of each node ,Then, select the k nodes with the highest scores to retain. In the process of selecting these k important nodes, firstly, Sort all nodes, select the k nodes with the highest scores, and obtain their indexes. Then, apply the tanh activation function to the importance scores of the selected k nodes, map these score values to the range of (-1, 1), and use them as the weight coefficients of the nodes. By multiplying the index features and weight coefficients element by element, the node features after node selection can be obtained. At the same time, the connection structure of the graph is reconstructed through the filtering operation, only the edges between the selected nodes are retained, and the corresponding edge indexes and attributes are updated to form a new adjacency matrix , the specific operations are as follows:

[0096] ,

[0097] ,

[0098] ,

[0099] ,

[0100] In the above formula and Represent the node feature matrix and adjacency matrix respectively, Represents a graph convolution operation with a layer output channel of 1, represents the importance score of the output node, The top-k operation is used to select the k node indexes with the highest scores. topk(.) represents the top-k operation, which is an operation that can select the most important k targets according to their importance. The feature matrix of the selected node is and the score after tanh activation Multiply element by element to get the new node feature matrix , Represents the new adjacency matrix formed.

[0101] In the contrastive mutual information feature selection, a contrastive mutual information maximization strategy based on structure perception is used. Mutual information is the core concept in information theory that quantifies the amount of information shared by two random variables. In contrastive mutual information feature selection, the focus is on the mutual information between the node importance score and the original node features. By maximizing this mutual information, it is ensured that the node importance score Y generated by graph convolution can accurately evaluate the importance of the original node. At the same time, the perception of the graph topology structure is enhanced through the contrastive learning method based on structure perception. InfoNCE is a contrastive learning method for estimating and maximizing mutual information. First, the node embedding representation Y, that is, the node score Y, is calculated by graph convolution, and then a contrastive mutual information estimation loss based on structure perception is designed based on InfoNCE. , this contrastive mutual information estimation loss can effectively quantify and maximize the mutual information between node embeddings and original node features, and is defined as follows:

[0102] ,

[0103] in is the graph convolutional embedding representation of the node, is the original node feature, It is a negative sample feature constructed based on node degree, and MLP is a multi-layer perceptron used to evaluate the mutual information between feature pairs.

[0104] The positive sample pairs are constructed in the following way: for each node i, the embedding representation generated by graph convolution is The corresponding original features Pairing forms positive sample pairs.

[0105] The negative sample construction adopts a sampling strategy based on the topological heterogeneity of the brain network: first, the degree values of all nodes in the graph are calculated, and the nodes are divided into a high-degree node set and a low-degree node set with the average degree as the threshold. For high-degree nodes, random sampling is performed from the low-degree node set as negative samples; for low-degree nodes, random sampling is performed from the high-degree node set as negative samples. The embedding representation generated by graph convolution is Paired with the negative samples constructed based on node degree to form negative sample pairs.

[0106] The final output of the contrastive mutual information feature selection is generated by further applying the GCN graph convolution layer to the selected k node features to perform residual graph convolution operation:

[0107] ,

[0108] in and Represents the new node feature matrix and adjacency matrix obtained after comparative mutual information feature selection, Represents the final output of contrastive mutual information feature selection, i.e. brain connection feature embedding .

[0109] S2.2, using the constructed multimodal feature extractor to process the two-dimensional vector Obtain phenotypic information feature embedding :

[0110] A multimodal feature extraction module is constructed to extract To extract more representative feature representations, the multimodal feature extraction module is implemented using a multi-layer perceptron (MLP) structure, which includes two hidden layers. Each layer combines linear transformation, batch normalization, ReLU activation function, and Dropout regularization technology to improve the generalization and expression capabilities of the model. The phenotypic feature extraction process can be expressed as follows:

[0111] ,

[0112] in, and Represent the original phenotypic features and the phenotypic features with more potential after extraction, and each hidden layer function Defined as:

[0113] ,

[0114] in, and They represent the trainable parameters of the i-th hidden layer respectively. BatchNorm represents the normalization layer, which helps to accelerate network convergence and improve training stability. The ReLU activation function enables the network to learn complex feature interactions and effectively alleviates the gradient vanishing problem in deep networks. The Dropout layer reduces the risk of the model over-relying on certain features by randomly inactivating neurons, and ultimately outputs more refined features of phenotypic information data. , Captures the most relevant features of phenotypic data for disease diagnosis, i.e., phenotypic information feature embedding .

[0115] like Figure 5 As shown, in step S3 of this embodiment, the brain connection feature is integrated into the gated fusion network and phenotypic information feature embedding , get the fusion features , as shown below:

[0116] Embedding brain connectivity features using gated fusion networks and phenotypic information feature embedding They are processed independently, mapped to the hidden space of the same dimension through their respective linear transformation layers, and the ReLU activation function is applied to enhance the nonlinear expression ability to obtain the transformed feature representation and , as shown below:

[0117] ,

[0118] ,

[0119] ,

[0120] in, and They represent the trainable parameters, and are the results of linear transformation and ReLU activation of brain functional connectivity features and phenotypic features, respectively. and Splicing is performed on the feature dimension to form a higher-dimensional joint feature representation H. Based on this spliced feature H, the network calculates a gate value gate through a gating unit containing a linear layer and a Sigmoid activation function. The Sigmoid function ensures that the gate value is between 0 and 1, indicating the fusion ratio of the two features:

[0121] ,

[0122] Among them, fused is the splicing feature H mentioned above. Finally, the network uses this adaptive gating value gate to perform weighted fusion of the features of the two channels to obtain the final fused feature representation:

[0123] ,

[0124] Among them, gate is the gate value, is the output fusion feature.

[0125] In step S4 of this embodiment, based on the fusion feature Construct a heterogeneous population graph, extract population-level features and input them into the classifier to obtain the diagnosis results of autism;

[0126] S4.1, construct heterogeneous group graph:

[0127] Heterogeneous group graph consists of node sets and edge sets Composition, expressed as , where the node set Each node in represents a subject, the total number of nodes is n, and the edge set Indicates the data collection site relationship between different subjects. According to the different collection site relationships of the subject data, the edges are divided into the same collection site edges. and different collection sites , the individual feature embedding obtained after each subject is processed by the individual graph model is used as the node feature of the group graph , where h is the brain connectivity feature embedding obtained by each subject after passing through the individual graph model and phenotypic information feature embedding The feature dimension of the fused features uses the following phenotypic text information data similarity score calculation method: for categorical variables, if the values of two individuals are the same, they are considered similar; for numerical variables, the similarity is judged based on whether the difference between the two values is less than the set threshold. Finally, this process generates a phenotypic similarity matrix , the phenotypic text information data similarity scores are used as the weights of the edges between nodes. It is a threshold function that controls the sparsity of the group graph by retaining only the edges with weights greater than the set threshold, excluding weak associations, and finally obtaining the adjacency matrix Used to construct a group graph and, at the same time, construct a mask matrix based on site information , further divide the edges into similar edges (same site) and heterogeneous edges (different sites), and finally, combine the adjacency matrix and the mask matrix And the fusion feature Y extracted by the individual graph model, a heterogeneous group graph is constructed.

[0128] S4.2, dual-channel information aggregation and adaptive feature fusion:

[0129] The dual-channel information aggregation and adaptive feature fusion network structure is as follows: Figure 6 As shown in the figure, the network consists of three layers, each of which consists of two parallel TransformerConv graph convolutions, which aggregate the information of neighbors of the same site and neighbors of different sites respectively, and fuse homogeneous features and heterogeneous features through an adaptive weight mechanism. The initial fusion weight of homogeneous features is set to 0.8, and the initial fusion weight of heterogeneous features is set to 0.2. In the subsequent information aggregation process, learnable feature fusion weights are used, and the fusion weights are dynamically adjusted in an adaptive manner during model training to aggregate homogeneous and heterogeneous information. The dual-channel information aggregation and adaptive feature fusion process is shown below:

[0130] ,

[0131] ,

[0132] ,

[0133] in, , as well as Respectively represent the input The node feature set, homogeneous edge set and heterogeneous edge set of the layer, GraphTransformer represents the information aggregation process of TransformerConv graph convolution, and denote the feature embeddings learned from the same-site and different-site neighbors, respectively. and It is The layer-trainable homogeneous and heterogeneous feature fusion weights are updated in each layer through a normalization operation so that their sum is 1, as shown below:

[0134] ,

[0135] ,

[0136] The output features of each layer are processed by batch normalization and LeakyReLU activation function, and then used as the input of the next two-channel information aggregation layer, and are saved as And cascade with the output features of the subsequent layers to obtain group-level features :

[0137] ,

[0138] ,

[0139] in Represent the output of each layer respectively, and finally, the group level features Input a fully connected layer classifier to obtain the diagnosis result of autism.

[0140] In order to verify the multimodal autism diagnostic method based on resting-state fMRI and phenotypic text information in this embodiment, the datasets used in the experiment in this embodiment are ABIDE datasets, specifically including the ABIDE I dataset and the ABIDE II dataset. The evaluation indicators used are accuracy ACC, sensitivity SEN, specificity SPE, area under the ROC curve AUC, and F1_score. Tables 1 and 2 respectively compare the performance of the method in this embodiment with the existing methods on the ABIDE I dataset and the ABIDE II dataset. OURS in the table shows the performance of the multimodal autism diagnostic method based on resting-state fMRI and phenotypic text information in this embodiment.

[0141] Table 1: Performance comparison on the ABIDE I dataset

[0142] Module ACC SEN SPE AUC F1 EV-GCN 84.96 85.17 87.61 88.10 86.23 MAMF-GCN 88.63 90.23 88.59 94.41 89.70 MMGL 89.88 91.42 88.63 89.58 90.14 OURS 94.95 96.80 92.80 97.48 95.35

[0143] Table 2: Performance comparison on the ABIDE II dataset

[0144] Module ACC SEN SPE AUC F1 K-means-SVM 61.61 65.47 57.74 61.61 62.36 SCNN 70.22 67.28 74.38 70.83 70.71 DFC-CBGRU 71.81 73.92 71.03 72.47 71.39 OURS 96.85 97.80 95.76 99.26 97.08

[0145] As can be seen from Tables 1 and 2, the multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information in this embodiment is superior to general diagnostic methods in performance.

[0146] To further illustrate the effectiveness of the multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information in this embodiment, we used t-distributed random neighbor embedding to map features into a two-dimensional space, and compared the embedding representations of samples before and after model learning. Figure 7 (a) and Figure 8 (a) shows the distribution of raw features in the ABIDE I and ABIDE II datasets, where triangles and squares represent HC and ASD samples, respectively. As can be seen from the visualization, the raw features of ASD and HC samples are chaotically mixed together, making it difficult to classify different samples based on the raw features. There is no clear dividing line between the two types of sample feature distributions, which indicates that classification based on their raw features is extremely challenging. Figure 7 (b) and Figure 8 (b) shows a visualization of the distribution of sample features after classification. After training with the multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information in this embodiment, the originally chaotic features are divided into two classes with a clear dividing line in the middle, which makes it easy to distinguish between autistic patients and healthy subjects.

[0147] In summary, the multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information in this embodiment first uses a pre-trained heterogeneous graph neural network model to obtain the output autism diagnosis result by using the brain resting-state functional magnetic resonance and phenotypic text information: obtaining the input individual's resting-state functional magnetic resonance and phenotypic text information; using resting-state functional magnetic resonance to construct a brain functional connection map; encoding the phenotypic text information into a two-dimensional vector; using a graph convolutional network and contrastive mutual information feature selection method to capture the high-order features of the brain functional connection map to obtain the individual brain connection feature embedding under the fMRI modality; using the constructed multimodal feature extraction module to extract features from the encoded two-dimensional vector to obtain to the phenotypic information feature embedding in the text modality; the individual brain connection feature embedding is fused with the phenotypic feature embedding through a gated fusion network to obtain a fused individual feature representation; based on the fused individual feature representation and a predefined phenotypic information similarity measurement method, a heterogeneous group graph is constructed, in which nodes represent individuals and edges represent specific relationships between individuals; dual-channel information aggregation and adaptive feature fusion are used on the heterogeneous group graph to learn the relationship between individuals and aggregate information, and finally group-level features are obtained to output autism diagnosis results. The present invention aims to use multimodal information for comprehensive analysis to improve the accuracy of autism diagnosis, and can be applied to fields such as neuroimaging analysis and intelligent medicine.

[0148] In addition, this embodiment also provides a multimodal autism diagnostic system based on resting-state fMRI and phenotypic text information, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the multimodal autism diagnostic method based on resting-state fMRI and phenotypic text information.

[0149] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instructions, and the computer program or instructions are programmed or configured to execute the multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information through a processor.

[0150] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information through a processor.

[0151] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products, and therefore, the present application may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions described in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0152] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal autism diagnostic method based on resting-state fMRI and phenotypic text information, characterized in that: The method includes using a pre-trained heterogeneous graph neural network model to obtain an output autism diagnosis result by using resting-state functional magnetic resonance imaging (fMRI) and phenotypic text information. The heterogeneous graph neural network model generates the autism diagnosis result by the following steps: S1: Obtain the resting-state functional magnetic resonance imaging (fMRI) and phenotypic text information of the input individual, construct a brain functional connectivity map, and encode the phenotypic text information into a two-dimensional vector. ; S2: Processing brain functional connectivity maps to obtain brain connectivity feature embedding , processing two-dimensional vectors Obtain phenotypic information feature embedding ; S3: Fusion of brain connectivity feature embeddings via a gated fusion network and phenotypic information feature embedding , get the fusion features ; S4: Based on fusion features Construct a heterogeneous population graph, extract population-level features, and input them into the classifier to obtain the diagnosis results of autism, including: S4.1, construct heterogeneous group graph: Each node of the heterogeneous population graph represents a subject. The edges are divided into the same collection site edges and different collection site edges according to the different collection sites of the subject data. The fusion features of each subject They are used as node features of the population graph, and the phenotypic data similarity scores are used as weights of the edges between nodes, thereby constructing a heterogeneous population graph; S4.2, dual-channel information aggregation and adaptive feature fusion extract group-level features for diagnostic classification: The dual-channel information aggregation consists of three layers, each of which is composed of two parallel TransformerConv graph convolutional networks, which respectively aggregate the information of neighbors at the same site and neighbors at different sites, and fuse homogeneous features with heterogeneous features through an adaptive weight mechanism. During the model training process, the fusion weights are dynamically adjusted in an adaptive manner to aggregate homogeneous and heterogeneous information to obtain group-level features; finally, the obtained group-level features are input into a fully connected layer classifier to obtain the diagnosis results of autism.

2. The multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information according to claim 1, characterized in that: In S1, the resting-state functional magnetic resonance imaging (fMRI) and phenotypic text information of the input individual are obtained, a brain functional connectivity map is constructed, and the phenotypic text information is encoded into a two-dimensional vector. , specifically including: S1.1, construct brain functional connectivity map based on input resting-state functional magnetic resonance imaging: Resting-state functional magnetic resonance imaging data were preprocessed, the brain was segmented into 200 regions, the average time series of each brain region were mapped and extracted, the Pearson correlation coefficient between the time series was calculated, and a functional connectivity matrix was obtained to construct a brain functional connectivity map; S1.2, encode phenotypic text information into a two-dimensional vector : The category text information in the phenotypic text information is one-hot encoded; the continuous text information is normalized, and the preprocessed phenotypic text information is encoded into a two-dimensional vector .

3. The multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information according to claim 1, characterized in that: In S2, the brain functional connectivity map is processed to obtain brain connectivity feature embedding , processing two-dimensional vectors Obtain phenotypic information feature embedding , specifically including: S2.

1. Extracting brain connectivity feature embedding from brain functional connectivity graph using graph convolutional network and contrastive mutual information feature selection : The graph convolutional network is a 2-layer GCN graph convolutional network layer. The contrastive mutual information feature selection is a mutual information feature selection method based on contrastive learning. The potential node features are extracted through the graph convolutional network, and then the contrastive mutual information feature selection method is used to select the most important node features as brain connection feature embedding ; S2.2, using the constructed multimodal feature extractor to process the two-dimensional vector Obtain phenotypic information feature embedding : A multimodal feature extraction module is used to extract the Extracting more representative phenotypic information feature embedding .

4. The multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information according to claim 1, characterized in that: In S3, brain connectivity feature embeddings are fused via a gated fusion network and phenotypic information feature embedding , get the fusion features , specifically including: Gated fusion network can embed the brain connection features according to the input and phenotypic information feature embedding The gating mechanism dynamically adjusts the fusion ratio of the two features based on their contribution to the diagnostic results, thereby processing information from different modalities more flexibly. When the features of a certain modality are more helpful for the diagnostic classification of the current sample, the gating mechanism will automatically give the modality a higher weight; otherwise, its influence will be reduced.

5. A multimodal autism diagnostic system based on resting-state fMRI and phenotypic text information, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information as described in any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instructions are programmed or configured to execute, through a processor, the multimodal autism diagnostic method based on resting-state fMRI and phenotypic text information as described in any one of claims 1 to 4.

7. A computer program product comprising a computer program or instructions, characterized in that The computer program or instructions are programmed or configured to execute, through a processor, the multimodal autism diagnostic method based on resting-state fMRI and phenotypic text information as described in any one of claims 1 to 4.

Images