Multi-mode autism diagnosis method and system based on resting-state fMRI and phenotypic text information, medium and product

Abstract

The invention discloses a multi-mode autism diagnosis method and system based on resting state fMRI and phenotypic text information, a medium and a product. FMRI and phenotypic text information are processed through a pre-trained heterogeneous graph neural network model to output a diagnosis result; the resting state fMRI is utilized to construct a brain function connection graph, and the phenotypic text information is coded into a two-dimensional vector; respectively extracting brain connection feature embedding of the fMRI mode and phenotype information feature embedding of the text mode; fusing the feature embedding in the two modes through a gating fusion network; and constructing a heterogeneous group diagram based on the fusion features and phenotypic information similarity, and obtaining group diagram features on the heterogeneous group diagram by using dual-channel information aggregation and adaptive feature fusion to output a diagnosis result. The invention aims to improve the accuracy of autism diagnosis by using multi-modal information, and can be applied to the fields of neuroimaging analysis, intelligent medical treatment and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and particularly relates to a multi-modal autism diagnosis method, system, medium and product based on resting-state fMRI and phenotypic text information. Background Art

[0002] Autism spectrum disorder is a serious and common public health problem globally. Autistic patients will show abnormalities in functions such as cognition, emotion, and behavior. Due to the lack of clear pathological mechanisms and reliable biomarkers for reference, the diagnosis results of doctors are easily affected by personal subjective judgments.

[0003] The brain is usually regarded as a complex network system, and functional magnetic resonance imaging technology has become an important tool for studying the functionality of the brain network in recent years. Brain functional connectivity analysis based on resting-state functional magnetic resonance imaging (rs-fMRI) has been widely used in the research of brain network disorders. These studies provide important evidence for understanding the neural mechanisms of autism spectrum disorder by extracting and analyzing the functional connectivity features between brain regions.

[0004] In recent years, graph neural networks, with their topological feature extraction ability and relational reasoning advantages, have provided a new paradigm for modeling functional brain connectivity networks. There are mainly two functional brain network modeling strategies in existing research: the individual graph method defines functional brain regions as nodes and the functional connectivity strength as edges; the group graph method defines subjects as nodes and constructs edges by calculating the similarity of phenotypic text information of subjects.

[0005] Existing methods face multiple bottlenecks. In the individual graph method, its nodes represent specific functional brain regions, and graph convolution is used to extract the brain functional connectivity features of a single subject, which can locate disease-related abnormal brain regions, but there are deficiencies in prediction accuracy; the group graph method uses phenotypic text information to calculate the similarity between subjects to construct a relational network, and extracts the relational features between groups through graph convolution, with relatively high accuracy, but its nodes only represent abstract individual features and it is difficult to locate specific functional brain regions, weakening the identification of biomarkers; using only single-modal brain functional connectivity features for diagnosis ignores that phenotypic text information also contains some features that are very important for diagnosis and cannot make full use of the comprehensive features of subjects; in addition, during the acquisition process of resting-state functional magnetic resonance data, due to significant differences in instrument equipment, data acquisition protocols, subject population characteristics, and preprocessing methods among different acquisition sites, the data shows obvious heterogeneity in dimensions such as quality, resolution, and signal-to-noise ratio. The existence of this data heterogeneity will have a certain impact on diagnosis and is likely to cause subjective judgment errors. Summary of the Invention

[0006] The objective of the present invention is to provide a multi-modal autism diagnosis method, system, medium, and product based on resting-state fMRI and phenotypic text information, so as to solve the problems existing in the above-mentioned prior art and reduce the decision-making risk of autism diagnosis.

[0007] To achieve the above objective, the present invention provides the following solutions:

[0008] The present invention provides a multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information, including the following steps:

[0009] S1: Obtain the brain resting-state functional magnetic resonance and phenotypic text information of the input individual, construct a brain functional connectivity map, and encode the phenotypic text information into a two-dimensional vector ;

[0010] S2: Process the brain functional connectivity map to obtain a brain connectivity feature embedding , process the two-dimensional vector to obtain a phenotypic information feature embedding ;

[0011] S3: Fusion the brain connectivity feature embedding and the phenotypic information feature embedding through a gated fusion network to obtain a fusion feature ;

[0012] S4: Based on the fusion feature construct a heterogeneous population graph, extract population-level features and input them into a classifier to obtain the diagnosis result of autism;

[0013] Further, in S1, obtain the brain resting-state functional magnetic resonance and phenotypic text information of the input individual, construct a brain functional connectivity map, and encode the phenotypic text information into a two-dimensional vector , specifically as follows;

[0014] S1.1, construct a brain functional connectivity map according to the input resting-state functional magnetic resonance;

[0015] Preprocess the resting-state functional magnetic resonance imaging (rs-fMRI) data, use the CC200 brain atlas template to divide the brain into 200 regions, map and extract the average time series of each brain region, and obtain the average time series of 200 brain regions based on blood oxygenation level dependence. For the processed time series , where N is the number of divided brain regions and T is the length of the time series (in milliseconds), calculate the Pearson correlation coefficient between the time series of each brain region and the time series of other brain regions to obtain a functional connectivity matrix to construct a brain functional connectivity map , where is a set of nodes, where each node represents a brain region. is a set of edges, which contains the functional connectivity information between each pair of brain regions.

[0016] First, apply Fisher's r-to-z transformation to the functional connectivity matrix to make the Pearson correlation coefficient distribution closer to a normal distribution. Each brain region serves as a node in the graph, and the node features are directly taken as the corresponding row vectors in the transformed functional connectivity matrix, representing the functional connectivity relationship between this brain region and all other brain regions. For edge construction, take the absolute value of the functional connectivity matrix, convert it into an initial fully connected graph, then remove self-loop connections and apply a threshold for filtering, only retaining the edges with a functional connectivity strength greater than the threshold. Finally, the edge weights are directly taken as the Pearson correlation coefficients between brain regions.

[0017] S1.2, Encode the phenotypic text information into a two-dimensional vector ;

[0018] Specifically, first perform one-hot encoding on the two types of categorical text information, i.e., site information and gender information; then normalize the continuous text information, including normalizing age and three IQ indicators (Verbal IQ, VIQ, Performance IQ, PIQ, and Full IQ, FIQ) by subtracting the mean and dividing by the standard deviation. Finally, the preprocessed phenotypic text information data is encoded into a two-dimensional vector , where represents the number of subjects, is the feature dimension of the phenotypic text information of the subjects.

[0019] Furthermore, in S2, process the brain functional connectivity graph to obtain brain connection feature embeddings , process the two-dimensional vector to obtain phenotypic information feature embeddings , as follows;

[0020] S2.1, Use a graph convolutional network and contrastive mutual information feature selection to extract brain connection feature embeddings from the brain functional connectivity graph ;

[0021] When using a graph convolutional network and contrastive mutual information feature selection to extract brain connection feature embeddings from the brain functional connectivity graph , the graph convolutional network is a 2-layer GCN graph convolutional network layer, and the contrastive mutual information feature selection is a mutual information feature selection method based on contrastive learning. Specifically, given node features , first use a GCN graph convolutional layer with 1 output channel to estimate the importance score of each node , Subsequently, the k nodes with the highest scores are selected and retained. During the process of selecting these k important nodes, first, according to the importance scores sort all the nodes. While selecting the k nodes with the highest scores, their indices index are obtained. Then, apply the tanh activation function to the importance scores of the selected k nodes to map these score values to the range (-1, 1) as the weight coefficients of the nodes. By multiplying the index features and the weight coefficients element-wise, the node features after node selection can be obtained , Meanwhile, reconstruct the connection structure of the graph through a filtering operation, only retain the edges between the selected nodes, and update the corresponding edge indices and attributes, thus forming a new adjacency matrix , The specific operations are as follows: , , , .

[0022] In the above formula and represent the node feature matrix and the adjacency matrix respectively, represents a graph convolution operation with an output channel of 1 for one layer, represents the importance scores of the output nodes, is the index of the k nodes with the highest scores selected through the top-k operation. topk(.) represents the top-k operation, which is an operation that can select the most important k targets according to importance; the feature matrix of the selected nodes is multiplied element-wise with the scores after being activated by tanh to obtain a new node feature matrix , represents the newly formed adjacency matrix.

[0023] In the process of contrastive mutual information feature selection, a structure-aware contrastive mutual information maximization strategy is used. Mutual information is a core concept in information theory that quantifies the amount of information shared by two random variables. In contrastive mutual information feature selection, the mutual information between the node importance scores and the original node features is focused on. By maximizing this mutual information, it is ensured that the node importance scores Y generated by graph convolution can accurately evaluate the importance of the original nodes. At the same time, the structure-aware contrastive learning method enhances the ability to perceive the graph topology. InfoNCE is a contrastive learning method used to estimate and maximize mutual information. In contrastive mutual information feature selection, first, calculate the node embedding representation Y through graph convolution, where Y is the node importance score, and then construct a structure-aware contrastive mutual information estimation loss based on InfoNCE , this contrast mutual information estimation loss can effectively quantify and maximize the mutual information between the node embedding Y and the original node features, and is defined as follows: ,

[0024] where is the graph convolutional embedding representation of the node, is the original node feature, is the negative sample feature constructed based on the node degree, and MLP is a multi-layer perceptron used to evaluate the mutual information between feature pairs.

[0025] Positive sample pairs are constructed in the following way: for each node i, pair its embedding representation generated by graph convolution with its corresponding original feature to form positive sample pairs.

[0026] Negative samples are constructed using a sampling strategy based on the topological heterogeneity of the brain network: first calculate the degree values of all nodes in the graph, and divide the nodes into a high-degree node set and a low-degree node set with the average degree as the threshold. For high-degree nodes, randomly sample from the low-degree node set as negative samples; for low-degree nodes, randomly sample from the high-degree node set as negative samples, and pair the embedding representation generated by graph convolution with its negative sample constructed based on the node degree to form negative sample pairs.

[0027] The final output of the contrast mutual information feature selection is generated by further applying the GCN graph convolutional network layer to the selected k node features for residual graph convolution operation: ,

[0028] where and represent the new node feature matrix and adjacency matrix obtained after contrast mutual information feature selection, represents the final output of the contrast mutual information feature selection, that is, the brain connection feature embedding .

[0029] S2.2. Use the constructed multi-modal feature extractor to process the two-dimensional vector to obtain the phenotypic information feature embedding :

[0030] A multi-modal feature extraction module is constructed to extract from Extract more representative feature representations. The multi-modal feature extraction module is implemented using a multi-layer perceptron (MLP) structure, which includes two hidden layers. Each layer combines linear transformation, batch normalization, ReLU activation function, and Dropout regularization techniques to improve the generalization ability and expressive power of the model. The phenotypic feature extraction process can be expressed as: ,

[0031] Among them, and represent the original phenotypic features and the more potentially powerful phenotypic features after extraction respectively. Each hidden layer function is defined as: ,

[0032] Among them, and represent the trainable parameters of the i-th hidden layer respectively. BatchNorm represents the normalization layer, which helps to accelerate network convergence and improve training stability. The ReLU activation function enables the network to learn complex feature interaction relationships and effectively alleviates the vanishing gradient problem in deep networks. The Dropout layer reduces the risk of the model over-relying on certain features by randomly deactivating neurons. Finally, a more refined phenotypic information feature of the phenotypic text information data is output , that is, the phenotypic information feature embedding , which captures the features most relevant to disease diagnosis in the phenotypic text information data.

[0033] Furthermore, in S3, the brain connection feature embedding and the phenotypic information feature embedding are fused through a gated fusion network to obtain the fused feature , as follows:

[0034] The gated fusion network independently processes the brain connection feature embedding and the phenotypic information feature embedding , maps them to a hidden space of the same dimension through their respective linear transformation layers, and applies the ReLU activation function to enhance the non-linear expression ability, obtaining the transformed feature representations and , as shown below: , , ,

[0035] Among them, and represent the trainable parameters respectively, and They are the results after the brain functional connectivity features and phenotypic features undergo linear transformation and ReLU activation. Subsequently, and are concatenated in the feature dimension to form a higher-dimensional joint feature representation H. Based on this concatenated feature H, the network calculates a gating value gate through a gating unit that includes a linear layer and a Sigmoid activation function. The Sigmoid function ensures that the gating value is between 0 and 1, representing the fusion ratio of the two features: ,

[0036] where fused is the concatenated feature H mentioned above. Finally, the network uses this adaptive gating value to weight-fuse the features of the two channels to obtain the final fused feature representation: ,

[0037] where gate is the gating value, is the output fused feature.

[0038] Furthermore, in S4, based on the fused feature a heterogeneous population graph is constructed, and the population-level features are extracted and input into the classifier to obtain the diagnosis result of autism, specifically as follows;

[0039] S4.1, Construct a heterogeneous population graph:

[0040] The heterogeneous population graph consists of a node set and an edge set and is denoted as . Among them, each node in the node set represents a subject, and the total number of nodes is n. The edge set represents the data collection site relationship between different subjects. According to the different data collection sites of the subjects, the edges are divided into edges with the same collection site and edges with different collection sites . The individual feature embeddings obtained after each subject is processed by the individual graph model are used as the node features of the population graph , where h is the feature dimension of the fused feature of the brain connectivity feature embedding and the phenotypic information feature embedding obtained by each subject after passing through the individual graph model. The following phenotypic information data similarity scoring calculation method is used: for categorical variables, if the values of two individuals are the same, they are considered to have similarity; for numerical variables, similarity is judged based on whether the difference between their values is less than the set threshold. Finally, this process generates a phenotypic similarity matrix , and the phenotypic information data similarity scores are used as the weights of the edges between the nodes. It is a threshold function that controls the sparsity of the population graph by retaining only the edges with weights greater than the set threshold, excluding weak associations, and finally obtaining the adjacency matrix It is used to construct the population graph. At the same time, a mask matrix is constructed by combining the site information The edges are further divided into homogeneous edges (same site) and heterogeneous edges (different sites). Finally, combined with the adjacency matrix and the mask matrix as well as the individual feature embeddings extracted from the individual graph model, a heterogeneous population graph is constructed.

[0041] S4.2, Dual-channel Information Aggregation and Adaptive Feature Fusion:

[0042] This network consists of three layers, each layer composed of two parallel TransformerConv graph convolutions, which respectively aggregate the information of the same-site neighbors and different-site neighbors, and fuse the homogeneous features and heterogeneous features through an adaptive weight mechanism. The initial fusion weight of the homogeneous features is set to 0.8, and the initial fusion weight of the heterogeneous features is set to 0.2. In the subsequent information aggregation process, learnable feature fusion weights are used to dynamically adjust the fusion weights in an adaptive manner during the model training process to aggregate homogeneous and heterogeneous information. The process of dual-channel information aggregation and adaptive feature fusion is as follows: , , ,

[0043] Among them, , and respectively represent the set of node features, the set of homogeneous edges, and the set of heterogeneous edges input to the L-th layer. GraphTransformer represents the information aggregation process of the TransformerConv graph convolution, and respectively represent the feature embeddings learned from the same-site and different-site neighbors, and are the trainable homogeneous and heterogeneous feature fusion weights of the L-th layer. Each layer is updated through a normalization operation to make their sum equal to 1, as follows: , ,

[0044] The output features of each layer are processed by Batchnorm normalization and the LeakyReLU activation function, and used as the input of the dual-channel information aggregation layer of the next layer. At the same time, they will be saved as and concatenated with the output features of the subsequent layers to obtain the population-level features : , ,

[0045] where respectively represent the outputs of each layer. Finally, the group-level features are input into a fully connected layer classifier to obtain the diagnosis result of autism.

[0046] In addition, the present invention also provides a multi-modal autism diagnosis system based on resting-state fMRI and phenotypic text information, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information.

[0047] In addition, the present invention also provides a computer-readable storage medium, in which a computer program or instruction is stored, and the computer program or instruction is programmed or configured to execute the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information through a processor.

[0048] In addition, the present invention also provides a computer program product, including a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information through a processor.

[0049] Compared with the prior art, the present invention mainly has the following advantages: combining the advantages of individual graphs and group graphs, achieving a relatively high diagnostic accuracy rate; at the same time, using a multi-modal feature extraction module to extract phenotypic information feature embeddings complementary to brain connection feature embeddings from phenotypic text information, and complementarily fusing feature embeddings of different modalities, making full use of multi-modal data and effectively improving the diagnostic accuracy rate; in addition, due to significant differences in instrument equipment, data acquisition protocols, subject population characteristics, and preprocessing methods among different acquisition sites during the collection process of resting-state functional magnetic resonance data, the data shows obvious heterogeneity in dimensions such as quality, resolution, and signal-to-noise ratio. To address this problem, a heterogeneous group graph is constructed based on the relationship of data acquisition sites, and a dual-channel information aggregation and adaptive feature fusion method is used to extract group-level features for classification, effectively solving the problem of data heterogeneity. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a schematic diagram of the basic process of the method according to an embodiment of the present invention;

[0051] Figure 2 is an algorithm framework diagram of the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information provided by an embodiment of the present invention;

[0052] Figure 3 For obtaining the brain connection feature embedding in the embodiments of the present invention and the phenotypic information feature embedding is a schematic diagram of the working principle;

[0053] Figure 4 is a schematic diagram of the working principle of the mutual information feature selection method for comparison in the embodiments of the present invention;

[0054] Figure 5 is a schematic diagram of the working principle of the gated fusion network model in the embodiments of the present invention;

[0055] Figure 6 is a schematic diagram of the working principle of the dual-channel information aggregation and adaptive feature fusion network in the embodiments of the present invention;

[0056] Figure 7 is a visualization diagram of the classification effect in the ABIDE I dataset in the embodiments of the present invention, where Figure 7 (a) represents the distribution of the original features in the ABIDE I dataset, Figure 7 (b) represents the distribution of the sample features after classification;

[0057] Figure 8 is a visualization diagram of the classification effect in the ABIDE II dataset in the embodiments of the present invention, where Figure 8 (a) represents the distribution of the original features in the ABIDE II dataset, Figure 8 (b) represents the distribution of the sample features after classification. Specific embodiments

[0058] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present invention.

[0059] The core of the present invention is to provide a multi-modal autism diagnosis method, system, medium and product based on resting-state fMRI and phenotypic text information to solve the problems existing in the existing method technologies.

[0060] To enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0061] Figure 1Schematic diagram of the basic process of a multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information provided by an embodiment of the present invention, including the following steps:

[0062] S1: Obtain the brain resting-state functional magnetic resonance and phenotypic text information of the input individual, construct a brain functional connectivity map, and encode the phenotypic text information into a two-dimensional vector ;

[0063] S2: Process the brain functional connectivity map to obtain brain connectivity feature embeddings , process the two-dimensional vector to obtain phenotypic information feature embeddings ;

[0064] S3: Fusion the brain connectivity feature embeddings and phenotypic information feature embeddings through a gated fusion network to obtain fusion features ;

[0065] S4: Based on the fusion features construct a heterogeneous population graph, extract population-level features and input them into a classifier to obtain the diagnosis result of autism.

[0066] Figure 2 Algorithm framework diagram of the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information provided by an embodiment of the present invention. The multi-modal autism diagnosis method provided by the present invention consists of two parts: an individual graph and a population graph. In the individual graph, a brain functional connectivity map of each subject is constructed, individual brain connectivity feature embeddings are extracted through two-layer GCN graph convolution, and an irrelevant feature removal method based on contrastive learning mutual information feature selection is adopted. To make full use of non-imaging phenotypic text information data, a multi-modal feature extraction module is introduced to extract features from the phenotypic text information data, and a gated fusion network is used to fuse the phenotypic information feature embeddings and individual brain connectivity feature embeddings as node features in the population graph. In the population graph, a heterogeneous population graph based on the relationship of data collection sites is constructed, with subjects as nodes, and the node relationship is described by same-site edges and different-site edges. The edge weights are calculated from the similarity of the phenotypic information data. Finally, population-level features are extracted through a dual-channel information aggregation and adaptive feature fusion method and sent to a fully connected layer to distinguish between healthy control groups and autism patients, i.e., HC and ASD.

[0067] As Figure 3 shown, in step S1 of this embodiment, the brain resting-state functional magnetic resonance and phenotypic text information of the input individual are obtained, a brain functional connectivity map is constructed using the resting-state functional magnetic resonance, and the phenotypic text information is encoded into a two-dimensional vector , specifically as follows;

[0068] S1.1. Construct a brain functional connectivity map based on the input resting-state functional magnetic resonance imaging:

[0069] Preprocess the resting-state functional magnetic resonance imaging (rs-fMRI) data, segment the brain into 200 brain regions using the CC200 brain atlas template, map and extract the average time series of each brain region, and obtain the average time series of 200 brain regions based on blood oxygenation level dependence. For the processed time series , where N is the number of brain regions and T is the length of the time series (in milliseconds), calculate the Pearson correlation coefficient between the time series of each brain region and the time series of other brain regions to obtain a functional connectivity matrix to construct a brain functional connectivity map , where is the set of nodes, and each node represents a brain region, is the set of edges, which contains the connection information between each brain region.

[0070] First, apply Fisher's r-to-z transformation to the functional connectivity matrix to make the Pearson correlation coefficient distribution closer to the normal distribution. Each brain region is used as a node in the graph, and the node features are directly represented by the corresponding row vector in the transformed functional connectivity matrix, indicating the functional connectivity between this brain region and all other brain regions. For the construction of edges, take the absolute value of the functional connectivity matrix, convert it into an initial fully connected graph, then remove the self-loop connections and apply a threshold for filtering, only retaining the edges with a functional connectivity strength greater than the threshold. Finally, the weight of the edge is directly the Pearson correlation coefficient between the brain regions.

[0071] S1.2. Encode the phenotypic text information into a two-dimensional vector :

[0072] Specifically, first perform one-hot encoding on the two types of categorical text information, site information and gender information; then normalize the continuous text information, including normalizing age and three intelligence quotient indicators (verbal intelligence quotient VIQ, performance intelligence quotient PIQ, and full intelligence quotient FIQ) by subtracting the mean and dividing by the standard deviation. Finally, the preprocessed phenotypic text information data is encoded into a two-dimensional vector , where represents the number of subjects, is the phenotypic feature dimension of the subject.

[0073] As Figure 3 shown, in step S2 of this embodiment, the brain functional connectivity map is processed to obtain brain connection feature embeddings , the two-dimensional vector is processed to obtain phenotypic information feature embeddings , specifically as follows;

[0074] S2.1. Extract brain connection feature embeddings from brain functional connectivity graphs using graph convolutional networks and contrastive mutual information feature selection :

[0075] Extract brain connection feature embeddings from brain functional connectivity graphs using graph convolutional networks and contrastive mutual information feature selection When doing so, the graph convolutional network is a 2-layer GCN graph convolutional network layer, and the contrastive mutual information feature selection is a mutual information feature selection method based on contrastive learning. As Figure 4 shown, specifically, given node features , first use a graph convolutional (GCN) layer with an output channel of 1 to estimate the importance score of each node . Subsequently, select and retain the k nodes with the highest scores. In the process of selecting these k important nodes, first sort all nodes according to the importance score . While selecting the k nodes with the highest scores, obtain their indices index. Then, apply the tanh activation function to the importance scores of the selected k nodes to map these score values to the range (-1, 1) as the weight coefficients of the nodes. By multiplying the index features and the weight coefficients element-wise, the node features after node selection can be obtained . At the same time, reconstruct the connection structure of the graph through a screening operation, only retain the edges between the selected nodes, and update the corresponding edge indices and attributes, thereby forming a new adjacency matrix . The specific operations are as follows: , , , ,

[0076] In the above formula and represent the node feature matrix and the adjacency matrix respectively, represents a graph convolutional operation with an output channel of 1 for one layer, represents the importance score of the output nodes, is the index of the k nodes with the highest scores selected through the top-k operation. topk(.) represents the top-k operation, which is an operation that can select the most important k targets according to importance; multiply the feature matrix of the selected nodes element-wise with the score after the tanh activation to obtain a new node feature matrix , represents the newly formed adjacency matrix

[0077] In the contrastive mutual information feature selection, a structure-aware contrastive mutual information maximization strategy is used. Mutual information is a core concept in information theory that quantifies the amount of information shared between two random variables. In contrastive mutual information feature selection, the mutual information between the node importance score and the original node features is focused on. By maximizing this mutual information, it is ensured that the node importance score Y generated by graph convolution can accurately evaluate the importance of the original nodes. At the same time, the ability to perceive the graph topology structure is enhanced through a structure-aware contrastive learning method. InfoNCE is a contrastive learning method used to estimate and maximize mutual information. First, the node embedding representation Y, that is, the node score Y, is calculated through graph convolution, and then a structure-aware contrastive mutual information estimation loss is designed based on InfoNCE. This contrastive mutual information estimation loss can effectively quantify and maximize the mutual information between the node embedding and the original node features, and is defined as follows: ,

[0078] where is the graph convolution embedding representation of the node, is the original node feature, is the negative sample feature constructed based on the node degree, and MLP is a multi-layer perceptron used to evaluate the mutual information between feature pairs.

[0079] The positive sample pairs are constructed in the following way: for each node i, the embedding representation generated by graph convolution for it is paired with its corresponding original feature to form positive sample pairs.

[0080] The negative samples are constructed using a sampling strategy based on the topological heterogeneity of the brain network: first, the degree values of all nodes in the graph are calculated, and the nodes are divided into a high-degree node set and a low-degree node set with the average degree as the threshold. For high-degree nodes, random samples are taken from the low-degree node set as negative samples; for low-degree nodes, random samples are taken from the high-degree node set as negative samples. The embedding representation generated by graph convolution is paired with its negative sample constructed based on the node degree to form negative sample pairs.

[0081] The final output of the contrastive mutual information feature selection is generated by further applying the GCN graph convolution layer to the selected k node features for residual graph convolution operations: ,

[0082] where and represent the new node feature matrix and adjacency matrix obtained after contrastive mutual information feature selection, Represents the final output of the contrastive mutual information feature selection, i.e., the brain connection feature embedding 。

[0083] S2.2, Process the two-dimensional vector using the constructed multi-modal feature extractor Obtain the phenotypic information feature embedding :

[0084] Constructed a multi-modal feature extraction module for extracting more representative feature representations from The multi-modal feature extraction module is implemented using a multi-layer perceptron (MLP) structure, including two hidden layers. Each layer combines linear transformation, batch normalization, ReLU activation function, and Dropout regularization techniques to improve the generalization ability and expressive ability of the model. The phenotypic feature extraction process can be expressed as: ,

[0085] Where and respectively represent the original phenotypic features and the more potentially capable phenotypic features after extraction. Each hidden layer function is defined as: ,

[0086] Where and respectively represent the trainable parameters of the i-th hidden layer. BatchNorm represents the normalization layer, which helps to accelerate network convergence and improve training stability. The ReLU activation function enables the network to learn complex feature interaction relationships and effectively alleviates the vanishing gradient problem in deep networks. The Dropout layer reduces the risk of the model over-relying on certain features by randomly deactivating neurons. The final output is a more refined feature of the phenotypic information data , captures the features most relevant to disease diagnosis in the phenotypic data, that is, the phenotypic information feature embedding 。

[0087] As Figure 5 shown, in step S3 of this embodiment, the brain connection feature embedding and the phenotypic information feature embedding are fused through a gated fusion network to obtain the fused feature as follows:

[0088] The gated fusion network fuses the brain connection feature embedding and the phenotypic information feature embedding Perform independent processing, map them to a hidden space of the same dimension through their respective linear transformation layers, and apply the ReLU activation function to enhance the non-linear expression ability to obtain the transformed feature representation and , specifically as follows: , , ,

[0089] Among them, and respectively represent trainable parameters, and are the results of the brain functional connectivity features and phenotypic features after linear transformation and ReLU activation respectively. Subsequently, and are concatenated in the feature dimension to form a joint feature representation H of a higher dimension. Based on this concatenated feature H, the network calculates a gating value gate through a gating unit containing a linear layer and a Sigmoid activation function. The Sigmoid function ensures that the gating value is between 0 and 1, representing the fusion ratio of the two features: ,

[0090] Among them, fused is the concatenated feature H mentioned above. Finally, the network uses this adaptive gating value gate to weight and fuse the features of the two channels to obtain the final fused feature representation: ,

[0091] Among them, gate is the gating value, is the output fused feature.

[0092] In step S4 of this embodiment, based on the fused feature construct a heterogeneous population graph, extract population-level features and input them into a classifier to obtain the diagnosis result of autism;

[0093] S4.1, Construct a heterogeneous population graph:

[0094] The heterogeneous population graph consists of a node set and an edge set and is denoted as where each node in the node set represents a subject, the total number of nodes is n, and the edge set represents the data collection site relationship between different subjects. According to the different data collection site relationships of the subjects, the edges are divided into same-site edges and different-site edges , the individual feature embeddings obtained after processing each subject through the individual graph model are used as the node features of the population graph , where h is the brain connection feature embedding obtained by each subject after passing through the individual graph model and the phenotypic information feature embedding For the feature dimension of the fused features, the following method for calculating the similarity score of phenotypic text information data is used: For categorical variables, if the values of two individuals are the same, they are considered similar; for numerical variables, similarity is judged based on whether the difference between their values is less than the set threshold. Finally, this process generates a phenotypic similarity matrix , and the phenotypic text information data similarity score is used as the weight of the edge between nodes. is a threshold function that controls the sparsity of the population graph by only retaining edges with weights greater than the set threshold, excluding weak associations, and finally obtaining the adjacency matrix is used to construct the population graph. At the same time, a mask matrix is constructed by combining the site information , further dividing the edges into homogeneous edges (same site) and heterogeneous edges (different sites). Finally, combining the adjacency matrix and the mask matrix and the fused feature Y extracted by the individual graph model, a heterogeneous population graph is constructed.

[0095] S4.2, Dual-channel Information Aggregation and Adaptive Feature Fusion:

[0096] The dual-channel information aggregation and adaptive feature fusion network structure is as Figure 6 shown. This network contains three layers, and each layer consists of two parallel TransformerConv graph convolutions, which respectively aggregate the information of neighbors at the same site and neighbors at different sites, and fuse the homogeneous features and heterogeneous features through an adaptive weight mechanism. The initial fusion weight of the homogeneous features is set to 0.8, and the initial fusion weight of the heterogeneous features is set to 0.2. In the subsequent information aggregation process, learnable feature fusion weights are used, and the fusion weights are dynamically adjusted in an adaptive manner during the model training process to aggregate homogeneous and heterogeneous information. The dual-channel information aggregation and adaptive feature fusion process is as follows: , , ,

[0097] Among them, , and respectively represent the input at the The node feature set, the set of isomorphic edges, and the set of heterogeneous edges of the layer. GraphTransformer represents the information aggregation process of TransformerConv graph convolution. and represent the feature embeddings learned from the same-site and different-site neighbors respectively. and are the trainable isomorphic and heterogeneous feature fusion weights of the th layer. Each layer is updated through a normalization operation so that their sum is 1, as shown below: , ,

[0098] After the output features of each layer are processed by batch normalization and the LeakyReLU activation function, they are used as the input of the two-channel information aggregation layer of the next layer, and will be saved as and concatenated with the output features of subsequent layers to obtain the population-level features : , ,

[0099] where represent the outputs of each layer respectively. Finally, the population-level features are input into a fully connected layer classifier to obtain the diagnosis result of autism.

[0100] To verify the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information in this embodiment, the dataset used in the experiment in this embodiment is the ABIDE dataset, specifically including the ABIDE I dataset and the ABIDE II dataset. The evaluation metrics used are accuracy ACC, sensitivity SEN, specificity SPE, the area under the ROC curve AUC, and F1_score. Tables 1 and 2 respectively show the performance comparison of the method in this embodiment with existing methods on the ABIDE I dataset and the ABIDE II dataset. OURS in the table represents the performance of the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information in this embodiment.

[0101] Table 1: Performance comparison on the ABIDE I dataset Module ACC SEN SPE AUC F1 EV-GCN 84.96 85.17 87.61 88.10 86.23 MAMF-GCN 88.63 90.23 88.59 94.41 89.70 MMGL 89.88 91.42 88.63 89.58 90.14 OURS 94.95 96.80 92.80 97.48 95.35

[0102] Table 2: Performance comparison on the ABIDE II dataset Module ACC SEN SPE AUC F1 K-means-SVM 61.61 65.47 57.74 61.61 62.36 SCNN 70.22 67.28 74.38 70.83 70.71 DFC-CBGRU 71.81 73.92 71.03 72.47 71.39 OURS 96.85 97.80 95.76 99.26 97.08

[0103] As can be seen from Table 1 and Table 2, the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information in this embodiment is superior to general diagnosis methods in terms of performance.

[0104] To further illustrate the effectiveness of the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information in this embodiment, t-distributed stochastic neighbor embedding was used to map the features into a two-dimensional space, and the embedded representations of the samples before and after model learning were compared. Figure 7 (a) and Figure 8 (a) shows the distribution of the original features in the ABIDE I and ABIDE II datasets, where triangles and squares are used to represent the two types of samples, HC and ASD, respectively. It can be seen from the visualization results that the original sample features of the two types, ASD and HC, are mixed up, and it is very difficult to classify different samples based on the original features. There is no clear and definite boundary line in the feature distributions of the two types of samples, which indicates that it is extremely challenging to classify based on their original features. Figure 7 (b) and Figure 8 (b) shows the visualization of the feature distribution of the samples after classification. After being trained by the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information in this embodiment, the original chaotic features are divided into two classes, and the boundary line in the middle is clear and definite, which makes it easy to distinguish autistic patients from healthy subjects.

[0105] In summary, the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information in this embodiment first uses a pre-trained heterogeneous graph neural network model for brain resting-state functional magnetic resonance and phenotypic text information to obtain the output autism diagnosis result: obtain the resting-state functional magnetic resonance and phenotypic text information of the input individual; construct a brain functional connectivity graph using the resting-state functional magnetic resonance; encode the phenotypic text information into a two-dimensional vector; use the graph convolutional network and the contrastive mutual information feature selection method to capture the high-order features of the brain functional connectivity graph to obtain the individual brain connectivity feature embedding in the fMRI modality; use the constructed multi-modal feature extraction module to extract features from the encoded two-dimensional vector to obtain the phenotypic information feature embedding in the text modality; fuse the individual brain connectivity feature embedding and the phenotypic feature embedding through a gated fusion network to obtain the fused individual feature representation; construct a heterogeneous population graph based on the fused individual feature representation and a predefined phenotypic information similarity measurement method, where the nodes in the graph represent individuals and the edges represent specific relationships between individuals; use dual-channel information aggregation and adaptive feature fusion on the heterogeneous population graph to learn the relationships between individuals and aggregate information, and finally obtain the population-level features to output the autism diagnosis result. The present invention aims to use multi-modal information for comprehensive analysis to improve the accuracy of autism diagnosis and can be applied to fields such as neuroimaging analysis and intelligent medicine.

[0106] In addition, this embodiment also provides a multimodal autism diagnosis system based on resting-state fMRI and phenotypic text information, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information.

[0107] In addition, this embodiment also provides a computer-readable storage medium, in which a computer program or instruction is stored, and the computer program or instruction is programmed or configured to execute the multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information through a processor.

[0108] In addition, this embodiment also provides a computer program product, including a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multimodal autism diagnosis method based on resting-state fMRI and phenotypic text information through a processor.

[0109] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one or more flows and / or blocks Figure 1 one or more blocks. These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one or more flows and / or blocks Figure 1The functions specified in one or more boxes. These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide for implementing in the process Figure 1 One process or more processes and / or boxes Figure 1 The steps of the functions specified in one box or more boxes.

[0110] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information, characterized in that, Including obtaining an autism diagnosis result by using a pre-trained heterogeneous graph neural network model for brain resting-state functional magnetic resonance and phenotypic text information. The steps for the heterogeneous graph neural network model to generate an autism diagnosis result are as follows: S1: Obtain the resting-state functional magnetic resonance imaging and phenotypic text information of the input individual, construct a brain functional connectivity map, and encode the phenotypic text information into a two-dimensional vector ; S2: Process the brain functional connectivity map to obtain brain connection feature embeddings , process the two-dimensional vector to obtain phenotypic information feature embeddings ; S3: Fuse the brain connection feature embedding and the phenotypic information feature embedding through the gated fusion network to obtain the fused feature. and the phenotypic information feature embedding , to obtain the fused feature ; S4: Based on the fused features Construct a heterogeneous population graph, extract the population hierarchical features, and input them into a classifier to obtain the diagnosis result of autism.

2. The multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information according to claim 1, wherein In S1, the resting-state functional magnetic resonance imaging and phenotypic text information of the input individual are obtained to construct a brain functional connectivity map, and the phenotypic text information is encoded into a two-dimensional vector , which specifically includes: S1.1, constructing a brain functional connectivity map based on the input resting-state functional magnetic resonance: Preprocessing the resting-state functional magnetic resonance imaging data, dividing the brain into 200 regions, mapping and extracting the average time series of each brain region, calculating the Pearson correlation coefficient between the time series, and obtaining a functional connectivity matrix to construct a brain functional connectivity map; S1.2, encode the phenotypic text information into a two-dimensional vector : One-hot encode the categorical text information in the phenotypic text information; normalize the continuous text information. The phenotypic text information after preprocessing is encoded as a two-dimensional vector .

3. The multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information according to claim 1, characterized in that In S2, the brain functional connectivity map is processed to obtain brain connection feature embeddings , and the two-dimensional vector is processed to obtain phenotypic information feature embeddings , specifically including: S2.

1. Extract brain connection feature embeddings from brain functional connectivity graphs using graph convolutional networks and contrastive mutual information feature selection : The graph convolutional network is a 2-layer GCN graph convolutional network layer. The contrastive mutual information feature selection is a mutual information feature selection method based on contrastive learning. It extracts potential node features through the graph convolutional network and then uses the contrastive mutual information feature selection method to screen the most important node features as brain connection feature embeddings ; S2.2, processing the two-dimensional vector by using the constructed multi-modal feature extractor Obtaining phenotypic information feature embedding : Utilize a multi-modal feature extraction module for extracting more representative phenotypic information feature embeddings from two-dimensional vectors therefrom .

4. The multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information according to claim 1, characterized in that In S3, the brain connection feature embedding and the phenotypic information feature embedding are fused through a gated fusion network to obtain a fused feature, specifically including: and the phenotypic information feature embedding , to obtain a fused feature , specifically including: The gated fusion network can embed according to the input brain connection feature embedding and phenotypic information feature embedding The contribution degree of the diagnostic result, dynamically adjust the fusion ratio of the two features, so as to more flexibly process information of different modalities. When the features of a certain modality are more helpful for the diagnosis and classification of the current sample, the gating mechanism will automatically assign a higher weight to this modality; otherwise, its influence will be reduced.

5. The multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information according to claim 1, characterized in that, In S4, based on the fused features Construct a heterogeneous population graph, extract population hierarchical features, input them into a classifier, and obtain the diagnosis result of autism, specifically including: S4.1, constructing a heterogeneous population graph: Each node in the heterogeneous population graph represents a subject. According to the different collection hospital sites of the subject data, the edges are divided into same-collection-site edges and different-collection-site edges. The fused features of each subject are used as the node features of the population graph, and the phenotypic data similarity scores are used as the weights of the edges between the nodes, thus constructing a heterogeneous population graph; S4.2, performing dual-channel information aggregation and adaptive feature fusion to extract population-level features for diagnostic classification: The dual-channel information aggregation consists of three layers, each layer composed of two parallel TransformerConv graph convolutional networks, which respectively aggregate the information of same-site neighbors and different-site neighbors, and fuse homogeneous and heterogeneous features through an adaptive weight mechanism. During the model training process, the fusion weights are dynamically adjusted in an adaptive manner to aggregate homogeneous and heterogeneous information, obtaining population-level features; finally, the obtained population-level features are input into a fully connected layer classifier to obtain an autism diagnosis result.

6. A multi-modal autism diagnosis system based on resting-state fMRI and phenotypic text information, comprising a microprocessor and a memory connected to each other, characterized in that, The microprocessor is programmed or configured to execute the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information according to any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instruction is programmed or configured to execute the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information according to any one of claims 1 to 5 through a processor.

8. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instruction is programmed or configured to execute the multi-modal autism diagnosis method based on resting-state fMRI and phenotypic text information according to any one of claims 1 to 5 through a processor.

Images