Depression state detection system based on multi-mode nuclear magnetic imaging map neural network

Abstract

The invention discloses a depressive state detection system based on a multi-mode nuclear magnetic imaging map neural network. The system comprises a brain data processing module, a feature extraction module and a classifier module. The brain data processing module is used for carrying out grid aggregation on the brain according to the input fMRI data to obtain a plurality of graph nodes, calculating the graph nodes and boundary values, and further completing the construction of a brain function connection graph; the feature extraction module analyzes the brain function connection diagram, extracts a feature vector H through a network layer and a pooling layer, and characterizes the features of brain function connection; and the classifier module trains a depression state classifier, and finally, the trained depression state classifier outputs the probability of the depression patient or the healthy individual according to the input feature vector H. According to the method, depression features in fMRI data are comprehensively explored by using the graph neural network and a self-attention mechanism, and support is provided for depression state detection. And by means of the domain classifier, the robustness and generalization ability of the depressive state classifier are improved.

Description

Technical Field

[0001] The present invention relates to the field of medical rehabilitation therapy, and in particular to a depression state detection system based on a multimodal nuclear magnetic resonance imaging neural network. Background Art

[0002] Depression, as a common mental disorder, is mainly manifested by symptoms such as persistent low mood and difficulty concentrating. This disease has the characteristics of a long onset period, repeated attacks and difficulty in complete cure, which brings serious harm to the patient's mental health. With the accelerated pace of modern society, the increase in life pressure and the combined influence of many other factors, the age distribution of patients with depression is becoming more and more extensive, and the total number of patients is showing a significant upward trend. Depression not only affects the patient's mood and cognitive function, but may also lead to a series of problems such as social disorders, decreased professional ability, and strained family relationships. These problems further aggravate the patient's psychological pressure, making the treatment and rehabilitation of depression more difficult.

[0003] Neuromodulation therapy is currently commonly used in clinical practice and provides an effective non-drug treatment option for patients with depression, especially for patients for whom drug therapy is ineffective or unsuitable. Neuromodulation therapy has the advantages of being non-invasive or minimally invasive, reducing the risk of surgery. It uses electrical, magnetic, optical or chemical means to regulate the activity of specific areas of the brain, thereby specifically treating certain brain dysfunction diseases. Transcranial Magnetic Stimulation (TMS) is a type of neuromodulation technology that is non-invasive and reversible. It uses the magnetic field generated by the coil to generate local induced currents in the cerebral cortex, regulate the excitability of cerebral cortical neurons, and thus affect the function of the human brain. At present, transcranial magnetic stimulation has been widely used in the study of brain function and the treatment of certain neuropsychiatric diseases, such as depression, schizophrenia, Parkinson's disease, epilepsy and migraine, and has good clinical application value and prospects.

[0004] With the continuous development of medical imaging technology, doctors and researchers can observe and study the structure and function of the brain without causing harm to the human body, improve the efficiency of diagnosis and the accuracy of treatment, and deal with the differences between individuals, and can make personalized targeted treatment. Nowadays, magnetic resonance imaging (MRI) technology has been widely used in the diagnosis and treatment of brain dysfunction. Structural magnetic resonance imaging (sMRI) can present the detailed brain structure information of individual patients more clearly, reduce the differences in anatomical levels between individuals, and achieve stereoscopic geometric positioning of the patient's stimulation target through three-dimensional brain modeling. Compared with the traditional stimulation target positioning based on standard brain areas, it can be more accurately positioned and quantified. Based on sMRI, the maximum current generated by the TMS coil can be controlled in a specified area. Functional magnetic resonance imaging (fMRI) is an imaging technology used to study brain function. It indirectly measures neural activity by detecting changes in blood flow when the brain is active. fMRI can be used to study the functional interactions and collaborations between different brain regions. fMRI can affect the neural activity of other parts of the brain that are functionally connected to the stimulated part, such as the deep subcortical areas that are not easily stimulated. fMRI is of great value in studying brain functional connections. It can help scientists and doctors understand how the brain processes information and the relationship between brain dysfunction and disease.

[0005] When analyzing fMRI data, the brain regions of individuals are often divided according to standard brain partitioning methods, and the regions divided with subjective factors are used as seed points or regions of interest (ROI) for the next step of connection analysis. This is very dependent on the professional knowledge and subjective experience of researchers, or the use of standard space registration, but this process does not take into account the differences between individuals. The brain is scanned using an MRI scanner to obtain a series of three-dimensional images in a time series. The image at each time point contains a snapshot of the brain. In fMRI data analysis, voxels can be regarded as tiny areas of the brain, and each voxel contains the signal intensity at a specific time. TMS is a non-invasive brain stimulation technology that generates a short magnetic field pulse by placing an electromagnetic coil on the scalp, thereby generating a tiny current in a specific area of ​​the brain to affect the activity of neurons. The size of the brain area affected by the TMS coil is approximately 5mm×5mm×5mm. In order to elucidate the functional mechanisms of the brain, researchers have used a variety of methods to explore and understand the working principles of the brain through fMRI. Many of them are based on graph neural network node construction technology with predefined brain maps. This method usually relies on brain region division standards such as automated anatomical labeling (AAL) and Brodmann areas to determine the nodes in the graph with standardized templates. However, this method may have limitations in explaining individual differences, dynamic changes, or high-resolution data (Depression state detection system based on physiological signal synchronization and multi-feature fusion (CN202310576947.6) Depression state recognition system based on multi-level feature fusion (CN202310761351.3)). Summary of the invention

[0006] The present invention focuses on the problem of depressive state detection based on graph neural networks of multimodal magnetic resonance imaging, and realizes the diagnosis of individual depression state through the graph neural network LSTM-DD-GAT at the individual level based on fMRI, thereby solving the problems raised in the above background technology.

[0007] Graph Neural Networks (GNNs), due to their unique architecture and ability to process graph-structured data, can identify sets of functionally similar or closely interacting regions in the brain by analyzing fMRI data, which helps reveal the modular organizational structure of the brain. The advantage of GNNs in processing fMRI data is that they can learn directly on the graph structure without converting the graph data into traditional grid or vector form. This direct processing method enables GNNs to retain the original topological structure information of the graph data, thereby more accurately capturing the complexity and dynamics of the brain's functional network. And they can adapt to the differences in brain structure between different individuals because they can be trained for each individual's brain map, thereby providing personalized analysis results. GNNs can effectively capture the complex patterns and relationships in this graph structure.

[0008] The present invention provides a depression state detection system based on a multimodal nuclear magnetic imaging graph neural network, and constructs a new network model architecture, namely LSTM-DD-GAT (LSTM-Domain Discriminator-Graph Attention Network). In the LSTM-DD-GAT model, fMRI data is first processed by a graph construction module, which determines the position and number of graph nodes according to the range of TMS coil stimulation, and the connection relationship between them is determined based on correlation analysis. The constructed graph is analyzed by a feature extraction module to reveal the potential neural mechanism related to depression. The extracted features are input into a domain classifier to identify and eliminate the feature differences between the source domain and the target domain. Finally, the individual's depressive state is predicted by a label classifier.

[0009] The purpose of the present invention is to grid the fMRI data based on the stimulation range of the TMS coil to construct a graph network. The graph neural network and self-attention mechanism are used to comprehensively explore the depressive features in fMRI data to provide support for depressive state detection. The domain classifier is then used to improve the robustness and generalization ability of the depressive state classifier.

[0010] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0011] A depression state detection system based on a multimodal nuclear magnetic resonance imaging neural network, comprising a brain data processing module, a feature extraction module and a classifier module;

[0012] Among them, the brain data processing module divides the brain into grids and aggregates them according to the input fMRI data to obtain multiple graph nodes, calculates the graph nodes and edge values, and then completes the construction of the brain functional connection map;

[0013] The feature extraction module analyzes the brain functional connection map and extracts the feature vector H through the network layer and the pooling layer to characterize the characteristics of the brain functional connection;

[0014] The classifier module trains the depression state classifier, and finally the trained depression state classifier outputs the probability of a depression patient or a healthy individual according to the input feature vector H.

[0015] Furthermore, in the brain data processing module, the fMRI data are first input and the brain is divided into three-dimensional grids with equal spacing matching the stimulation range of the TMS coil;

[0016] The voxels of each grid are aggregated into graph nodes, and the long short-term memory network (LSTM) is used to extract time series information from them to capture the dynamic changes between graph nodes; then the functional connectivity between graph nodes is calculated, and the correlation between graph nodes is calculated based on the BOLD signal time series of voxels between different graph nodes using correlation analysis methods such as the Pearson correlation coefficient or the Spearman rank correlation coefficient or the partial correlation coefficient or the phase synchronization method. When the voxel time series of two graph nodes show statistically significant correlation, by setting a threshold τ, only the edges between graph nodes are established for the graph node pairs whose correlation exceeds the threshold, forming a functional connection graph; the final output functional connection graph data includes the connection information between graph nodes and correlation edges, thereby providing a basis for the subsequent LSTM-DD-GAT model to analyze depression-related neural activities.

[0017] Furthermore, the size of each grid was determined according to the specific purpose of the study and the stimulation characteristics of the TMS coil;

[0018] The size of the grid should match the stimulation range of the TMS coil to ensure that the details of the neural activity related to depression are captured, so as to better facilitate subsequent analysis of auxiliary treatment; for example, assuming that the stimulation range of the TMS coil is 10 mm in diameter, and it is hoped to capture the neural activity within the stimulation area, each grid can be divided into 5x5x5 mm cubes. This size ensures that there is sufficient resolution within the stimulation area to detect the activity of the brain area, while not being over-divided, resulting in excessive calculations.

[0019] After the grid is divided, the voxels (i.e., volume pixels) in each grid are aggregated into a graph node; this graph node represents the collective characteristics of the voxels in the grid area, including the local neural activity pattern. Since the TMS stimulation area and treatment conditions are related to the stimulation range of the TMS coil, by analyzing the stimulation area under appropriate grid division, the neural activity of the corresponding stimulation area after the stimulation can be better analyzed and detected, avoiding the situation of analyzing too many unnecessary areas or ignoring the information of necessary areas.

[0020] Furthermore, the feature extraction module includes a double-layer graph attention network layer GATs, a graph pooling layer, and multiple groups of graph attention network GAT (Graph Attention Networks) layers and graph pooling layers connected in sequence, wherein the double-layer graph attention network layer GATs includes two layers of graph attention network GAT layers connected in sequence;

[0021] The two-layer graph attention network layer GATs receives the functional connection graph data, including the feature vectors of the graph nodes and the edge weights between the graph nodes, extracts the local neighborhood information, captures the local features of the graph nodes and some global features, and outputs the updated functional connection graph data to a graph pooling layer. The graph pooling layer downsamples the graph and outputs the reduced complexity graph representation data to the readout layer and the graph attention network GAT layer in the first group of multiple groups of graph attention network GAT layers and graph pooling layers, extracts the local neighborhood information, and outputs the updated functional connection graph data to the graph pooling layer in the first group, further processes and refines the features, and helps to recover by aggregating the features. The important local features are repeated or reconstructed to maintain the balance between local and global information, while allowing the graph nodes with higher importance screened out by the graph pooling layer to transmit information. The graph pooling layer in the first group downsamples the graph and outputs the graph representation data with reduced complexity to the readout layer and the graph attention network GAT layer in the second group. And so on. Each group of graph attention network GAT layers and graph pooling layers extracts the local neighborhood information of the nodes and refines the node features group by group. The graph pooling layer in the last group outputs the graph representation data with reduced complexity to the readout layer. The readout layer finally integrates the input data and outputs a feature vector of fixed length, which represents the global information of the entire graph.

[0022] Furthermore, in the GAT layer of the graph attention network, a self-attention mechanism is introduced to assign different importance weights to each graph node in the brain functional connection map; this mechanism calculates the importance score of each graph neighbor node to the target node, and these scores reflect the correlation between the neighbor node and the target node; based on these scores, the GAT layer of the graph attention network can weight the features of the aggregated neighbor nodes, so as to better capture the complex node relationships in the process of information integration; this enables the GAT layer of the graph attention network to flexibly focus on the neighbor nodes that are more important to the target node, thereby improving the accuracy of data processing;

[0023] In the GAT layer of the graph attention network, the self-attention mechanism is introduced as follows:

[0024] For a graph node v and its graph neighbor node u, the attention score e vu It can be calculated by the following formula:

[0025]

[0026] Among them, h v and h u Represent the feature vectors of graph node v and graph neighbor node u respectively, W1 is a learnable weight matrix, a is another set of learnable parameters, || represents the concatenation of vectors, N(v) represents the set of all graph neighbor nodes of graph node v; LeakyReLU is an activation function that allows gradients to be transferred in negative regions by providing a small non-zero slope for negative input values.

[0027] After calculating the attention score, the feature vector of each graph node v’s graph neighbor node u is updated using the following formula:

[0028] h′ u =σ(∑ k∈N(v) α vu Wlq k );

[0029] Among them, h′ u is the feature vector of the updated graph neighbor node u, α vu is the above attention score e vu The normalized attention score, σ is the nonlinear activation function, and W2 is the learnable weight matrix;

[0030] Furthermore, in the graph pooling layer, a graph pooling method based on the self-attention mechanism is used to calculate the attention score of the node, so as to distinguish which nodes should be retained and which should be discarded; a graph pooling technology combining multiple self-attention mechanisms is used to evaluate the importance of the node, and the attention score is determined by the graph convolutional network based on the relationship between the full graph information and the node features;

[0031] In the multiple groups of graph attention network (GAT) layers and graph pooling layers, the graph pooling layer first obtains the self-attention score of each graph node through graph convolution and activation function processing; secondly, according to the sorting of self-attention scores, the top k graph nodes with the highest scores are retained to form a mask; finally, the features of the specified graph nodes and the topological information between these graph nodes are retained according to the mask, as follows:

[0032] X is the node representation of the graph node after passing through the graph attention network GAT layer, and the first attention score Z1 is calculated using graph convolution:

[0033]

[0034] Where A is the adjacency matrix, is the adjacency matrix with self-connection added, yes The degree matrix, I is the identity matrix, Θ1 is the t×1 parameter;

[0035] Use the full image information to calculate the second attention score Z2. First, use the average method to obtain the feature expression of the full image information:

[0036]

[0037] Where n is the number of graph nodes, u i is the graph node representation obtained by processing the i-th graph node through the graph attention network GAT layer, Θ2 is a t×t parameter matrix, and the attention weight of the graph node is determined by the result of the dot product operation using the full graph information and the graph node information. After obtaining the attention weight of each graph node, the full graph information can be obtained by weighted averaging the node features:

[0038]

[0039]

[0040] Therefore, the total attention score Z for each graph node is:

[0041]

[0042] Keep a portion of the nodes in the input graph based on the total attention score:

[0043]

[0044] Among them, k is the pooling ratio, which determines the number of graph nodes that need to be retained. Top-Rank returns the index of the topk nodes based on the total attention score Z, and Zmask is the mask of the retained nodes.

[0045] Furthermore, the readout layer is responsible for aggregating the features of all graph nodes to form a global graph representation, converting high-dimensional node features into lower-dimensional global feature representations for classification tasks. In essence, it is actually a pooling function, but the readout layer aggregates node embeddings into graph embeddings, and its output can be directly used for graph-level classification tasks. The additive pooling operation has superior expression ability compared to mean pooling and maximum pooling, so the additive pooling operation is used to mine the depressive features in fMRI data, as follows:

[0046]

[0047] in, and are the graph embedding and node embedding matrices of the readout layer corresponding to the lth group of GAT layers and graph pooling layers, respectively, and the feature H of the final input classifier is the concatenation of the features of all readout layers, that is,

[0048] The main task of the readout layer is to aggregate the features of each node to generate a global graph representation for subsequent classification tasks. The specific operation is to add and pool the embedded features of all nodes and summarize them into a global feature vector. This additive pooling method has been shown to be superior to mean pooling and maximum pooling in capturing the expression features of different graph structures. The output of the readout layer is a low-dimensional global feature representation that can effectively reflect the features in fMRI data.

[0049] Furthermore, in the classifier module, domain classifiers and depressive state classifiers are constructed; due to the differences in fMRI sample acquisition conditions, there are distribution inconsistencies between different data sets. In the transfer learning framework, the data set with labels is called the source domain, and the data set without labels is the target domain; therefore, the labels of the source domain and the target domain are:

[0050]

[0051] The domain classifier uses a fully connected layer to linearly transform the input feature H and reduce its dimension to a two-dimensional space. The data points in this space are used to distinguish samples from the source domain and the target domain. The Sigmoid function is then used for binary classification, that is, to identify whether the sample comes from the source domain or the target domain:

[0052]

[0053] In the domain classifier, 1 represents the target domain and 0 represents the source domain. It is the parameter matrix of the fully connected layer of the domain classifier; during the back-propagation process, the gradient of the domain classification loss of the domain classifier is automatically negated before being back-propagated to the parameters of the feature extractor, thereby achieving adversarial loss.

[0054] The purpose of the depression state classifier is to analyze the input feature H to identify whether the feature data belongs to a depressed patient or a healthy individual. The input of the depression state classifier is the data obtained by the domain classifier through the fully connected layer to perform linear transformation and dimensionality reduction mapping on the input feature H to a two-dimensional space, and output the probability of each category (depressed state or normal state). The labels corresponding to depression and health are:

[0055]

[0056] The depression state classifier outputs the probability of each category through a fully connected layer and a Sigmoid function:

[0057]

[0058] In the depression state classifier, 1 represents the depression state, and 0 represents the normal state. is the parameter matrix of the fully connected layer of the depression state classifier; the output of the depression state classifier is the probabilities of depression state (marked as 1) and normal state (marked as 0); these probabilities reflect the model's confidence that the input data belongs to a certain state.

[0059] Furthermore, in the classifier module, the existing overall data set is input, and the overall data set includes source domain (training set) data and target domain (test set) data, where the source domain data is represented by X S , the target domain data is represented by X T In the classifier module, the core goal of training is to minimize the loss function L of the depression state classifier C And maximize the domain classifier loss function L D , the gradient descent method is used to optimize the depression state classifier and the domain classifier, and the parameters of the depression state classifier and the domain classifier are adjusted through back propagation to reduce the overall loss function, as follows:

[0060]

[0061] The optimization process is as follows:

[0062]

[0063] Among them, W is the parameter of the feature extraction module; by minimizing the loss function of the depression state classifier, the depression state classifier can make the prediction result as close to the true label as possible; at the same time, by maximizing the loss function of the domain classifier, the goal is to make the source domain of the sample unable to be effectively identified.

[0064] The goal of the domain classifier is to maximize the domain classification loss and confuse the target domain data with the source domain data, but the goal of the depression state classifier is to minimize the depression state classification loss and achieve accurate classification of depression state. The depression state classifier and the domain classifier compete with each other during the training process to achieve a balance between the depression state classification loss and the domain classification loss. The purpose of the domain classifier and the depression state classifier is to enhance the generalization ability of the depression state classifier and ensure that the depression state classifier can accurately predict the depression state.

[0065] Compared with the prior art solutions, the advantages of the technical solution of the present invention are as follows:

[0066] The clinical diagnosis of depression mainly relies on the patient's self-report and the doctor's clinical experience. The selection of TMS targets also relies heavily on the doctor's professional experience. Moreover, fMRI data is based on the standard brain partitioning method to divide the brain regions of individuals, and the connection analysis of the regions divided with subjective factors is highly dependent on the professional knowledge and subjective experience of researchers.

[0067] In order to solve the above problems, the present invention grids the fMRI data based on the stimulation range of the TMS coil to construct a graph network. The graph neural network and the self-attention mechanism are used to comprehensively explore the depressive features in the fMRI data to provide support for the detection of depressive states. The robustness and generalization ability of the model are improved by using the domain classifier. The present invention proposes a depressive state detection system based on a multimodal nuclear magnetic imaging graph neural network, constructs an LSTM-DD-GAT network model architecture (including four modules), realizes the determination of the individual's depressive state, and constructs a brain function map network structure based on the voxel level by analyzing fMRI data. This structure can capture the functional connection relationship between different regions in the brain. GAT can effectively process graph structure data, and extract global features related to the depressive state by learning the local neighborhood information of the node. Through the training and verification of the LSTM-DD-GAT model, the evaluation of the individual's depressive state is realized.

[0068] 1. Compared with traditional diagnosis and treatment schemes, the brain data processing module of the present invention utilizes the characteristics of TMS coil stimulation to grid the fMRI data, making the analysis and diagnosis results more suitable for TMS stimulation treatment.

[0069] 2. The LSTM-DD-GAT network model architecture in the depression state detection system mentioned in the present invention improves the model's ability to process time series signals through the LSTM layer, and also reveals the potential time series relationship in fMRI data. It also reduces the difference between the source domain and target domain data distribution through the introduction of domain classifiers, enhances the model's generalization ability to different domains, and improves adaptability.

[0070] 3. In the specific processing of feature extraction, the present invention proposes a feature extraction module that combines multi-layer GAT, Graph Pooling layer and readout layer, which can highlight the local topological structure and node features of the graph, and determine the attention score by combining the relationship between the whole graph information and the node features, so that the feature extraction module can fully extract and utilize the representation ability of the whole graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a schematic diagram of the structure of a brain data processing module in an embodiment of the present invention;

[0072] Figure 2 It is a structural schematic diagram of a feature extraction module in an embodiment of the present invention;

[0073] Figure 3 Schematic diagram of the structure of the classifier module in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the specific implementation of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0075] Example:

[0076] A depression state detection system based on a multimodal nuclear magnetic resonance imaging neural network, comprising a brain data processing module, a feature extraction module and a classifier module;

[0077] Among them, the brain data processing module divides the brain into grids and aggregates them according to the input fMRI data to obtain multiple graph nodes, calculates the graph nodes and edge values, and then completes the construction of the brain functional connection map;

[0078] The feature extraction module analyzes the brain functional connection map and extracts the feature vector H through the network layer and the pooling layer to characterize the characteristics of the brain functional connection;

[0079] The classifier module trains the depression state classifier, and finally the trained depression state classifier outputs the probability of a depression patient or a healthy individual according to the input feature vector H.

[0080] like Figure 1 As shown, in the brain data processing module, fMRI data is first input and the brain is divided into three-dimensional grids with equal spacing that matches the stimulation range of the TMS coil;

[0081] The voxels of each grid are aggregated into graph nodes, and the long short-term memory network (LSTM) is used to extract time series information from them to capture the dynamic changes between graph nodes. Then, the functional connectivity between graph nodes is calculated. Based on the BOLD signal time series of voxels between different graph nodes, the correlation analysis method is used to calculate the correlation between graph nodes. When the voxel time series of two graph nodes show statistically significant correlation, a threshold τ is set to establish edges between graph nodes only for graph node pairs whose correlation exceeds the threshold, forming a functional connection graph. The final output functional connection graph data includes the connection information between graph nodes and correlation edges, thereby providing a basis for the subsequent LSTM-DD-GAT model to analyze depression-related neural activities.

[0082] In one embodiment, the correlation analysis method is a method such as Pearson correlation coefficient, Spearman rank correlation coefficient, partial correlation coefficient or phase synchronization;

[0083] Furthermore, the size of each grid was determined according to the specific purpose of the study and the stimulation characteristics of the TMS coil;

[0084] The size of the grid should match the stimulation range of the TMS coil to ensure that the details of the neural activity related to depression are captured, so as to better facilitate subsequent analysis of auxiliary treatment; for example, assuming that the stimulation range of the TMS coil is 10 mm in diameter, and it is hoped to capture the neural activity within the stimulation area, each grid can be divided into 5x5x5 mm cubes. This size ensures that there is sufficient resolution within the stimulation area to detect the activity of the brain area, while not being over-divided, resulting in excessive calculations.

[0085] After the grid is divided, the voxels (i.e., volume pixels) in each grid are aggregated into a graph node; this graph node represents the collective characteristics of the voxels in the grid area, including the local neural activity pattern. Since the TMS stimulation area and treatment conditions are related to the stimulation range of the TMS coil, by analyzing the stimulation area under appropriate grid division, the neural activity of the corresponding stimulation area after the stimulation can be better analyzed and detected, avoiding the situation of analyzing too many unnecessary areas or ignoring the information of necessary areas.

[0086] like Figure 2 As shown, the feature extraction module includes a double-layer graph attention network layer GATs, a graph pooling layer, and multiple groups of graph attention network GAT (Graph Attention Networks) layers and graph pooling layers connected in sequence, and the double-layer graph attention network layer GATs includes two layers of graph attention network GAT layers connected in sequence;

[0087] The two-layer graph attention network layer GATs receives the functional connection graph data, including the feature vectors of the graph nodes and the edge weights between the graph nodes, extracts the local neighborhood information, captures the local features of the graph nodes and some global features, and outputs the updated functional connection graph data to a graph pooling layer. The graph pooling layer downsamples the graph and outputs the reduced complexity graph representation data to the readout layer and the graph attention network GAT layer in the first group of multiple groups of graph attention network GAT layers and graph pooling layers, extracts the local neighborhood information, and outputs the updated functional connection graph data to the graph pooling layer in the first group, further processes and refines the features, and helps to recover by aggregating the features. The important local features are repeated or reconstructed to maintain the balance between local and global information, while allowing the graph nodes with higher importance screened out by the graph pooling layer to transmit information. The graph pooling layer in the first group downsamples the graph and outputs the graph representation data with reduced complexity to the readout layer and the graph attention network GAT layer in the second group. And so on. Each group of graph attention network GAT layers and graph pooling layers extracts the local neighborhood information of the nodes and refines the node features group by group. The graph pooling layer in the last group outputs the graph representation data with reduced complexity to the readout layer. The readout layer finally integrates the input data and outputs a feature vector of fixed length, which represents the global information of the entire graph.

[0088] Furthermore, in the GAT layer of the graph attention network, a self-attention mechanism is introduced to assign different importance weights to each graph node in the brain functional connection map; this mechanism calculates the importance score of each graph neighbor node to the target node, and these scores reflect the correlation between the neighbor node and the target node; based on these scores, the GAT layer of the graph attention network can weight the features of the aggregated neighbor nodes, so as to better capture the complex node relationships in the process of information integration; this enables the GAT layer of the graph attention network to flexibly focus on the neighbor nodes that are more important to the target node, thereby improving the accuracy of data processing;

[0089] In the GAT layer of the graph attention network, the self-attention mechanism is introduced as follows:

[0090] For a graph node v and its graph neighbor node u, the attention score e vu It can be calculated by the following formula:

[0091]

[0092] Among them, h v and h u Represent the feature vectors of graph node v and graph neighbor node u respectively, W1 is a learnable weight matrix, a is another set of learnable parameters, || represents the concatenation of vectors, N(v) represents the set of all graph neighbor nodes of graph node v; LeakyReLU is an activation function that allows gradients to be transferred in negative regions by providing a small non-zero slope for negative input values.

[0093] After calculating the attention score, the feature vector of each graph node v’s graph neighbor node u is updated using the following formula:

[0094] h′ u =σ(∑ k∈N(v) α vu Wlq k );

[0095] Among them, h′ u is the feature vector of the updated graph neighbor node u, α vu is the attention score e vu The normalized attention score, σ is the nonlinear activation function, and W2 is the learnable weight matrix.

[0096] Furthermore, in the graph pooling layer, a graph pooling method based on the self-attention mechanism is used to calculate the attention score of the node, so as to distinguish which nodes should be retained and which should be discarded; a graph pooling technology combining multiple self-attention mechanisms is used to evaluate the importance of the node, and the attention score is determined by the graph convolutional network based on the relationship between the full graph information and the node features;

[0097] In the multiple groups of graph attention network (GAT) layers and graph pooling layers, the graph pooling layer first obtains the self-attention score of each graph node through graph convolution and activation function processing; secondly, according to the sorting of self-attention scores, the top k graph nodes with the highest scores are retained to form a mask; finally, the features of the specified graph nodes and the topological information between these graph nodes are retained according to the mask, as follows:

[0098] X is the node representation of the graph node after passing through the graph attention network GAT layer, and the first attention score Z1 is calculated using graph convolution:

[0099]

[0100] Where A is the adjacency matrix, is the adjacency matrix with self-connection added, yes The degree matrix, I is the identity matrix, Θ1 is the t×1 parameter;

[0101] Use the full image information to calculate the second attention score Z2. First, use the average method to obtain the feature expression of the full image information:

[0102]

[0103] Where n is the number of graph nodes, u i is the graph node representation obtained by processing the i-th graph node through the graph attention network GAT layer, Θ2 is a t×t parameter matrix, and the attention weight of the graph node is determined by the result of the dot product operation using the full graph information and the graph node information. After obtaining the attention weight of each graph node, the full graph information can be obtained by weighted averaging the node features:

[0104]

[0105]

[0106] Therefore, the total attention score Z for each graph node is:

[0107]

[0108] Keep a portion of the nodes in the input graph based on the total attention score:

[0109]

[0110] Where k is the pooling ratio, which determines the number of graph nodes that need to be retained. Top-Rank returns the index of the topk nodes based on Z, and Zmask is the mask of the retained nodes.

[0111] Furthermore, the readout layer is responsible for aggregating the features of all graph nodes to form a global graph representation, converting high-dimensional node features into lower-dimensional global feature representations for classification tasks. In essence, it is actually a pooling function, but the readout layer aggregates node embeddings into graph embeddings, and its output can be directly used for graph-level classification tasks. The additive pooling operation has superior expression ability compared to mean pooling and maximum pooling, so the additive pooling operation is used to mine the depressive features in fMRI data, as follows:

[0112]

[0113] in, and are the graph embedding and node embedding matrices of the readout layer corresponding to the lth group of GAT layers and graph pooling layers, respectively, and the feature H of the final input classifier is the concatenation of the features of all readout layers, that is,

[0114] The main task of the readout layer is to aggregate the features of each node to generate a global graph representation for subsequent classification tasks. The specific operation is to add and pool the embedded features of all nodes and summarize them into a global feature vector. This additive pooling method has been shown to be superior to mean pooling and maximum pooling in capturing the expression features of different graph structures. The output of the readout layer is a low-dimensional global feature representation that can effectively reflect the features in fMRI data.

[0115] In one embodiment, Figure 3 As shown in the figure, in the classifier module, domain classifiers and depression state classifiers are constructed; due to the differences in fMRI sample acquisition conditions, there are distribution inconsistencies between different data sets. In the transfer learning framework, the data set with labels is called the source domain, and the data set without labels is the target domain. Therefore, the labels of the source domain and the target domain are:

[0116]

[0117] The domain classifier uses a fully connected layer to linearly transform the input feature H and reduce its dimension to a two-dimensional space. The data points in this space are used to distinguish samples from the source domain and the target domain. The Sigmoid function is then used for binary classification, that is, to identify whether the sample comes from the source domain or the target domain:

[0118]

[0119] In the domain classifier, 1 represents the target domain and 0 represents the source domain. It is the parameter matrix of the fully connected layer of the domain classifier; during the back-propagation process, the gradient of the domain classification loss of the domain classifier is automatically negated before being back-propagated to the parameters of the feature extractor, thereby achieving adversarial loss.

[0120] The purpose of the depression state classifier is to analyze the input feature H to identify whether the feature data belongs to a depressed patient or a healthy individual. The input of the depression state classifier is the data obtained by the domain classifier through the fully connected layer to perform linear transformation and dimensionality reduction mapping on the input feature H to a two-dimensional space, and output the probability of each category (depressed state or normal state). The labels corresponding to depression and health are:

[0121]

[0122] The depression state classifier outputs the probability of each category through a fully connected layer and a Sigmoid function:

[0123]

[0124] In the depression state classifier, 1 represents the depression state, and 0 represents the normal state. is the parameter matrix of the fully connected layer of the depression state classifier; the output of the depression state classifier is the probabilities of depression state (marked as 1) and normal state (marked as 0); these probabilities reflect the model's confidence that the input data belongs to a certain state.

[0125] Furthermore, in the classifier module, the existing overall data set is input, and the overall data set includes source domain (training set) data and target domain (test set) data, where the source domain data is represented by X S , the target domain data is represented by X T In the classifier module, the core goal of training is to minimize the loss function L of the depression state classifier C And maximize the domain classifier loss function L D , the gradient descent method is used to optimize the depression state classifier and the domain classifier, and the parameters of the depression state classifier and the domain classifier are adjusted through back propagation to reduce the overall loss function, as follows:

[0126]

[0127] The optimization process is as follows:

[0128]

[0129] Among them, W is the parameter of the feature extraction module; by minimizing the loss function of the depression state classifier, the depression state classifier can make the prediction result as close to the true label as possible; at the same time, by maximizing the loss function of the domain classifier, the goal is to make the source domain of the sample unable to be effectively identified.

[0130] The goal of the domain classifier is to maximize the domain classification loss and confuse the target domain data with the source domain data, but the goal of the depression state classifier is to minimize the depression state classification loss and achieve accurate classification of depression state. The depression state classifier and the domain classifier compete with each other during the training process to achieve a balance between the depression state classification loss and the domain classification loss. The purpose of the domain classifier and the depression state classifier is to enhance the generalization ability of the depression state classifier and ensure that the depression state classifier can accurately predict the depression state.

[0131] The preferred embodiments of the present application disclosed above are only used to help understand the present invention and its core ideas. For those skilled in the art, according to the ideas of the present invention, there will be changes in specific application scenarios and implementation operations, and this specification should not be interpreted as limiting the present invention. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. A depression state detection system based on multimodal nuclear magnetic imaging neural network, characterized in that: It includes a brain data processing module, a feature extraction module and a classifier module; Among them, the brain data processing module divides the brain into grids and aggregates them according to the input fMRI data to obtain multiple graph nodes, calculates the graph nodes and edge values, and then completes the construction of the brain functional connection map; The feature extraction module analyzes the brain functional connection map and extracts the feature vector H through the network layer and the pooling layer to characterize the characteristics of the brain functional connection; The classifier module trains the depression state classifier, and finally the trained depression state classifier outputs the probability of a depression patient or a healthy individual according to the input feature vector H.

2. According to claim 1, a depression state detection system based on multimodal nuclear magnetic imaging neural network is characterized in that: In the brain data processing module, fMRI data are first input and the brain is divided into three-dimensional grids with equal spacing that matches the stimulation range of the TMS coil; The voxels of each grid are aggregated into graph nodes, and time series information is extracted from them using a long short-term memory network to capture the dynamic changes between graph nodes. Then the functional connectivity between graph nodes is calculated. Based on the BOLD signal time series of voxels between different graph nodes, the correlation between graph nodes is calculated using a correlation analysis method. When the voxel time series of two graph nodes show statistically significant correlation, a threshold τ is set to establish edges between graph nodes only for graph node pairs whose correlation exceeds the threshold, thus forming a functional connection graph. The functional connection graph data finally output includes the connection information between graph nodes and correlation edges. The size of each grid is determined according to the specific purpose of the study and the stimulation characteristics of the TMS coil; The size of the grid should match the stimulation range of the TMS coil to ensure that details of neural activity relevant to depression are captured; After meshing, the voxels in each mesh are aggregated into a graph node; This graph node represents the collective characteristics of the voxels within the grid area, including the local neural activity pattern.

3. A depression state detection system based on multimodal nuclear magnetic imaging neural network according to claim 1, characterized in that: The feature extraction module includes a double-layer graph attention network layer GATs, a graph pooling layer, and multiple groups of graph attention network GAT layers and graph pooling layers connected in sequence, wherein the double-layer graph attention network layer GATs includes two layers of graph attention network GAT layers connected in sequence; The two-layer graph attention network layer GATs receives functional connection graph data, including feature vectors of graph nodes and edge weights between graph nodes, extracts local neighborhood information, and outputs updated functional connection graph data to a graph pooling layer. The graph pooling layer downsamples the graph and outputs graph representation data with reduced complexity to the readout layer and the graph attention network GAT layer in the first group of multiple groups of graph attention network GAT layers and graph pooling layers, extracts local neighborhood information, outputs updated functional connection graph data to the graph pooling layer in the first group, the graph pooling layer in the first group downsamples the graph, outputs graph representation data with reduced complexity to the readout layer and the graph attention network GAT layer in the second group, and so on. Each group of graph attention network GAT layers and graph pooling layers extracts local neighborhood information of nodes and refines node features group by group. The graph pooling layer in the last group outputs graph representation data with reduced complexity to the readout layer. The readout layer finally integrates the input data and outputs a feature vector of a fixed length representing the global information of the entire graph.

4. A depression state detection system based on multimodal nuclear magnetic imaging neural network according to claim 3, characterized in that: In the GAT layer of the graph attention network, the self-attention mechanism is introduced as follows: For a graph node v and its graph neighbor node u, the attention score e vu Calculated by the following formula: Among them, h v and h u Represent the feature vectors of graph node v and graph neighbor node u respectively, W1 is a learnable weight matrix, a is another set of learnable parameters, || represents the concatenation of vectors, and N(v) represents the set of all graph neighbor nodes of graph node v; LeakyReLU is an activation function that allows gradients to propagate in negative regions by providing a small non-zero slope for negative input values; After calculating the attention score, the feature vector of each graph node v’s graph neighbor node u is updated using the following formula: h′ u =σ(∑ k∈N(v) a vu W2h k ); Among them, h′ u is the feature vector of the updated graph neighbor node u, α vu is the attention score e vu The normalized attention score, σ is the nonlinear activation function, and W2 is the learnable weight matrix.

5. A depression state detection system based on multimodal nuclear magnetic imaging neural network according to claim 4, characterized in that: In the multiple groups of graph attention network (GAT) layers and graph pooling layers, the graph pooling layer first processes the graph convolution and activation function to obtain the self-attention score of each graph node; Secondly, according to the sorting of self-attention scores, the top k graph nodes with the highest scores are retained to form a mask; finally, the features of the specified graph nodes and the topological information between these graph nodes are retained according to the mask, as follows: X is the node representation of the graph node after passing through the graph attention network GAT layer, and the first attention score Z1 is calculated using graph convolution: Where A is the adjacency matrix, is the adjacency matrix with self-connection added, yes The degree matrix, I is the identity matrix, Θ1 is the t×1 parameter; Use the full image information to calculate the second attention score Z2. First, use the average method to obtain the feature expression of the full image information: Where n is the number of graph nodes, u i is the graph node representation obtained by processing the i-th graph node through the graph attention network GAT layer, Θ2 is a t×t parameter matrix, and the attention weight of the graph node is determined by the result of the dot product operation using the full graph information and the graph node information. After obtaining the attention weight of each graph node, the full graph information is obtained by weighted averaging the node features: Therefore, the total attention score Z for each graph node is: Keep a portion of the nodes in the input graph based on the total attention score: Among them, k is the pooling ratio, which determines the number of graph nodes that need to be retained. Top-Rank returns the index of the topk nodes based on the total attention score Z, and Zmask is the mask of the retained nodes.

6. A depression state detection system based on multimodal nuclear magnetic imaging neural network according to claim 5, characterized in that: The readout layer is responsible for aggregating the features of all graph nodes to form a global graph representation, converting high-dimensional node features into lower-dimensional global feature representations for classification tasks. In essence, it is actually a pooling function, but the readout layer aggregates node embeddings into graph embeddings, and its output can be directly used for graph-level classification tasks. The addition pooling operation is used, as follows: in, and are the graph embedding and node embedding matrices of the readout layer corresponding to the lth group of GAT layers and graph pooling layers, respectively, and the feature H of the final input classifier is the concatenation of the features of all readout layers, that is, 7. The depression state detection system based on multimodal nuclear magnetic imaging neural network according to claim 1 is characterized in that: In the classifier module, domain classifiers and depression state classifiers are constructed; the dataset with labels is called the source domain, and the dataset without labels is the target domain. The labels of the source domain and target domain are: The domain classifier uses a fully connected layer to linearly transform the input feature H and reduce its dimension to a two-dimensional space. The data points in this space are used to distinguish samples from the source domain and the target domain. The Sigmoid function is then used for binary classification, that is, to identify whether the sample comes from the source domain or the target domain: In the domain classifier, 1 represents the target domain and 0 represents the source domain. It is the parameter matrix of the fully connected layer of the domain classifier; during the back-propagation process, the gradient of the domain classification loss of the domain classifier is automatically negated before being back-propagated to the parameters of the feature extractor, thereby achieving adversarial loss.

8. A depression state detection system based on multimodal nuclear magnetic imaging neural network according to claim 7, characterized in that: The purpose of the depression state classifier is to analyze the input feature H to identify whether the feature data belongs to a depressed patient or a healthy individual. The input of the depression state classifier is the data obtained by the domain classifier through the fully connected layer to perform linear transformation and dimensionality reduction mapping on the input feature H to a two-dimensional space, and output the probability of each category (depressed state or normal state). The labels corresponding to depression and health are: The depression state classifier outputs the probability of each category through a fully connected layer and a Sigmoid function: In the depression state classifier, 1 represents the depression state, and 0 represents the normal state. It is the parameter matrix of the fully connected layer of the depression state classifier; the output of the depression state classifier is the probability of depression state and normal state.

9. A depression state detection system based on multimodal nuclear magnetic imaging neural network according to claim 8, characterized in that: In the classifier module, the existing overall data set is input, which includes source domain (training set) data and target domain (test set) data, where the source domain data is represented by X S , the target domain data is represented by X T In the classifier module, the core goal of training is to minimize the loss function L of the depression state classifier C And maximize the domain classifier loss function L D , the gradient descent method is used to optimize the depression state classifier and the domain classifier, and the parameters of the depression state classifier and the domain classifier are adjusted through back propagation to reduce the overall loss function, as follows:

10. A depression state detection system based on multimodal nuclear magnetic imaging neural network according to claim 9, characterized in that: In the classifier module, the optimization process is as follows: Among them, W is the parameter of the feature extraction module; by minimizing the loss function of the depression state classifier, the depression state classifier can make the prediction result as close to the true label as possible; at the same time, by maximizing the loss function of the domain classifier, the goal is to make the source domain of the sample unable to be effectively identified.

Images