Prediction model construction method and device based on semantic modeling and dynamic graph neural network

Abstract

The invention provides a prediction model construction method and device based on semantic modeling and a dynamic graph neural network, and relates to the field of image processing. According to rich information representation contained in a brain region template, a unified and structured brain region semantic description system is constructed through LLM assistance, a new normal form is provided for brain network semantic modeling, and the knowledge interpretability of a brain map is enhanced; the richness and expression ability of brain network expression are remarkably improved by fusing semantic information of brain region structures and functions through a semantic enhanced brain network graph; a node redistribution mechanism is introduced on the basis of a traditional GNN framework, the node redistribution mechanism can adaptively reconstruct a graph structure according to node features, redundant connection is effectively removed, key areas are highlighted, and the generalization ability and training efficiency of a model are improved; according to the model provided by the invention, semantic modeling and functional connection are combined, the abnormal mode of the brain network can be more accurately captured, and a more efficient and accurate intelligent means is provided for auxiliary classification of brain region detection data.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for constructing a prediction model based on semantic modeling and dynamic graph neural network. Background Art

[0002] Major Depressive Disorder (MDD), the second-leading cause of disability worldwide, has long relied on subjective assessments based on the Diagnostic and Statistical Manual of Mental Disorders (DSM) symptom scale for diagnosis, lacking objective biomarkers. In current clinical practice, doctors rely on a combination of patient self-report, behavioral observations, and exclusionary tests to make a diagnosis. However, this model is susceptible to interference from individual characteristics and comorbidities, resulting in a misdiagnosis rate of up to 30%-40%.

[0003] Functional Magnetic Resonance Imaging (fMRI) indirectly reflects temporal changes in neural activity by detecting blood-oxygen-level dependent (BOLD) signals. As a key measure of brain activity, the BOLD signal is a key component of constructing functional connectivity networks (FC). Furthermore, in assisting the diagnosis of complex mental disorders such as MDD, patients exhibit dynamic and abnormal changes in functional connectivity. Therefore, FC is often used as a key input for computer-assisted diagnosis. Constructing FC networks typically involves spatially partitioning the whole-brain BOLD signal based on predefined brain region templates. These partitioned regions serve as network nodes, and the correlations between them are calculated to determine edge weights. However, traditional methods often treat each brain region as a single, static node, ignoring its underlying multidimensional structural, functional, and disease-related information. This results in limited representational capacity of the brain network, which in turn affects the effectiveness of discriminating against major depressive disorder. At the same time, the division standards between different brain region templates are inconsistent, and there is a lack of systematic and structured brain region function description text, which further limits the deep modeling of brain region semantic features.

[0004] With the widespread application of large language models (LLMs) in healthcare, their powerful language understanding and knowledge extraction capabilities offer a promising approach for semantic modeling of brain regions. LLMs generate functional descriptions of brain regions, combined with key concept extraction and semantic feature encoding methods, allowing for the introduction of richer contextual information for each node, thereby enhancing the expressive power of brain network modeling. However, existing work remains limited in integrating semantic information with graph-brain networks.

[0005] At the same time, as an important progress in the field of graph structure modeling in recent years, Graph Neural Network (GNN) has been widely used in brain network analysis tasks. GNN effectively captures the connection relationship between brain regions and the interaction of node features through the information aggregation mechanism of adjacency structure, and has good feature extraction capabilities. It is especially suitable for the fusion modeling of complex brain map structures and multimodal features. However, existing research focuses on the brain network of static nodes. There is still a lack of a unified framework and effective strategy for how to effectively embed semantically enhanced brain region node information and dynamically learn graph structures to achieve feature recognition of major depression. This limits the use of semantic information of brain region nodes in the process of brain network construction in graph neural networks. The graph structure cannot be dynamically updated according to individual characteristics, and it is difficult to meet the requirements of high-expression and high-interpretation models for accurate diagnosis of MDD. Summary of the Invention

[0006] The present invention aims to utilize large language models and graph neural networks to provide a graph neural network model based on semantic modeling and node redistribution. By combining fMRI data and brain region template features, the modeling information of the brain network graph is enriched. At the same time, the node redistribution mechanism is used to dynamically reconstruct the graph structure during the learning process of the graph neural network, thereby improving the classification accuracy of fMRI signal data.

[0007] The present invention uses a large language model for semantic modeling and a node redistribution mechanism to reconstruct the brain network graph during the training process of the GNN, thereby achieving efficient feature learning of the brain network. The graph neural network based on semantic modeling and node redistribution described in the present invention includes a brain region node semantic representation module, a semantically enhanced brain network graph, a node redistribution graph neural network learning module, and a graph feature extraction and prediction module. The brain region node semantic representation module uses LLM to extract and model the structural and functional information of the brain region, generate a brain region semantic description with a unified standard, and embed the description information through a text encoder to obtain the semantic similarity between brain regions. The semantically enhanced brain network graph generates semantic edges between brain regions based on the above-mentioned semantic similarity, and at the same time uses the functional connection strength of each brain region with other brain regions as a node feature to construct a brain network that integrates semantic information. The node redistribution graph neural network learning module introduces a node redistribution mechanism on the basis of the traditional GNN framework, dynamically adjusts the connection structure of the graph according to the feature similarity between nodes, thereby compressing redundant connections, highlighting key information and improving the expression ability of the graph structure. The graph feature extraction and prediction module fuses the node features in the graph through average pooling and maximum pooling strategies, and then completes the classification and prediction tasks of brain functional magnetic resonance imaging data through a multi-layer perceptron.

[0008] The method for constructing a prediction model based on semantic modeling and dynamic graph neural network described in the present invention comprises the following steps:

[0009] Step 1: Semantic representation of brain region nodes. A large language model is used to semantically model the anatomical structure and functional attributes of each brain region based on domain knowledge. The large language model is then used to extract representative terms or keywords from the brain region descriptions as the core concept semantic representations of the brain region. The core concept semantic representations are input into a text encoder to obtain semantic embedding vectors, and the similarity between the semantic embedding vectors is then calculated.

[0010] Step 2: Construct a brain network diagram based on semantic enhancement; obtain a functional magnetic resonance imaging (fMRI) signal dataset and calculate the functional connectivity values between brain regions; use semantic similarity as side information and the functional connectivity values between brain regions as node features to construct a brain network diagram; the functional connectivity values between brain regions are generated by calculating the correlation of BOLD signals between each brain region using the Pearson correlation coefficient, and the BOLD signals are extracted from the preprocessed fMRI signals according to the brain region template.

[0011] Step 3: Construct a dynamic graph neural network. The dynamic graph neural network includes two layers of graph neural networks. A node reallocation mechanism is added between the two layers of graph neural networks. The node reallocation mechanism dynamically reconstructs the graph structure according to the node feature representation; the node features are updated through the dynamic graph neural network.

[0012] Step 4: Graph feature extraction and prediction: Aggregate the updated node features based on a dual-channel graph-level pooling strategy to obtain a graph-level representation, which is then input into a multi-layer perceptron classifier to obtain the final prediction result.

[0013] Step 5: Optimize the model using classification loss to obtain a trained prediction model.

[0014] The dynamic graph neural network is specifically as follows:

[0015] The input of the first-layer graph neural network is the semantically enhanced brain network graph, and its output corresponds to the input of the node reallocation mechanism. The input of the second-layer graph neural network is the brain network graph simplified by the node reallocation mechanism.

[0016] The node reallocation mechanism is as follows:

[0017] Suppose the node feature matrix in the graph output by the first graph neural network is Where N represents the number of nodes, and D represents the feature dimension of each node;

[0018] Based on the mapping of features to groups, the group assignment matrix is obtained The expression is as follows:

[0019] M=Softmax(XW)

[0020] Among them, each row in the group assignment matrix M represents the normalized assignment probability of a node among G groups. is a learnable weight matrix;

[0021] For each node i, the group with the highest probability of being assigned is determined to be the final group. j , the expression is as follows:

[0022]

[0023] Among them, M ij is the mapping probability of node i to group j in the group assignment matrix, The actual group assignment probability for node i, g ij Is to use Gumbel-Softmax to M ij The noise after resampling, exp is the exponential function, log is the logarithmic function, and τ is the temperature parameter;

[0024] After completing the group division of the nodes, the original graph is restructured according to the group assignment results: the nodes belonging to the same group are aggregated to construct a simplified graph structure. In this simplified graph structure, each group is regarded as a new node, and the connection relationship between groups is determined based on the original connection between nodes in the group.

[0025] The present invention also provides a prediction device based on semantic modeling and dynamic graph neural network, comprising:

[0026] Brain Region Node Semantic Representation Module: This module uses a large language model to semantically model the anatomical structure and functional attributes of each brain region based on domain knowledge. It then uses the large language model to extract representative terms or keywords from the brain region descriptions as the core concept semantic representations of the brain region. The core concept semantic representations are input into a text encoder to obtain semantic embedding vectors, and the similarity between the semantic embedding vectors is then calculated.

[0027] Semantic-enhanced brain network graph construction module: acquires functional magnetic resonance imaging (fMRI) signal data and calculates the functional connectivity values between brain regions; uses semantic similarity as side information and the functional connectivity values between brain regions as node features to construct a brain network graph;

[0028] Node reallocation graph neural network learning module: This module includes two layers of graph neural networks, between which a node reallocation mechanism is added. The node reallocation mechanism dynamically reconstructs the graph structure based on the node feature representation; node features are updated through the dynamic graph neural network.

[0029] Graph feature extraction and prediction module; based on the dual-channel graph-level pooling strategy, the updated node features are aggregated to obtain a graph-level representation, which is then input into the multi-layer perceptron classifier to obtain the final prediction result.

[0030] The beneficial effects of the present invention are as follows:

[0031] LLM is used to assist in building a unified and structured semantic description system for brain regions, providing a new paradigm for brain network semantic modeling and enhancing the knowledge interpretability of brain maps.

[0032] The integration of semantic information of brain region structure and function significantly improves the richness and expressiveness of brain network representation;

[0033] The node reallocation mechanism can adaptively reconstruct the graph structure based on node characteristics, effectively removing redundant connections, highlighting key areas, and improving the model's generalization ability and training efficiency;

[0034] The proposed model combines semantic modeling and functional connectivity, which can more accurately capture abnormal patterns of brain networks and provide a more efficient and accurate intelligent means for data processing, analysis and prediction of functional brain magnetic resonance imaging of mental illness. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a model building framework for graph neural networks based on semantic modeling and node redistribution.

[0036] Figure 2 It is the generation process and instance of the semantic representation of brain area nodes.

[0037] Figure 3 It is the training and prediction process of graph neural networks based on semantic modeling and node redistribution. DETAILED DESCRIPTION

[0038] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings and examples so that those skilled in the art can better understand the present invention.

[0039] like Figure 1 As shown, this embodiment provides a prediction device based on semantic modeling and dynamic graph neural network, including:

[0040] Brain Region Node Semantic Representation Module: This module uses a large language model to semantically model the anatomical structure and functional attributes of each brain region based on domain knowledge. It then uses the large language model to extract representative terms or keywords from the brain region descriptions as the core concept semantic representations of the brain region. The core concept semantic representations are input into a text encoder to obtain semantic embedding vectors, and the similarity between the semantic embedding vectors is then calculated.

[0041] Semantic-enhanced brain network graph construction module: acquires functional magnetic resonance imaging (fMRI) signal data and calculates the functional connectivity values between brain regions; uses semantic similarity as side information and the functional connectivity values between brain regions as node features to construct a brain network graph;

[0042] Node reallocation graph neural network learning module: This module includes two layers of graph neural networks, between which a node reallocation mechanism is added. The node reallocation mechanism dynamically reconstructs the graph structure based on the node feature representation; node features are updated through the dynamic graph neural network.

[0043] Graph feature extraction and prediction module; based on the dual-channel graph-level pooling strategy, the updated node features are aggregated to obtain a graph-level representation, which is then input into the multi-layer perceptron classifier to obtain the final prediction result.

[0044] Next, the device is further explained by combining the construction method of the prediction model based on semantic modeling and dynamic graph neural network.

[0045] like Figure 3 As shown, in a specific embodiment of the present invention, the method for constructing a prediction model based on semantic modeling and dynamic graph neural network is as follows:

[0046] S1: Brain region node semantic representation module: Figure 2 As shown in the figure, the process of semantically modeling the anatomical structure and functional attributes of each brain region based on domain knowledge using a large language model (LLM) includes two main steps: brain region description generation and brain region key concept extraction.

[0047] In the brain region description generation stage, the name data in the brain region template is obtained. In this embodiment, the Anatomical Automatic Labeling (AAL) template is used. By designing a standardized Prompt instruction, the LLM output is limited to the content of the structure and function description of the brain region, thereby generating a unified and standardized brain region semantic text.

[0048] In the brain region key concept extraction stage, LLM is further used to extract the top 10 representative terms or keywords from the above-generated brain region descriptions as the core concept semantic representation of the brain region. For example, the structural and functional description of the brain region "amygdala" is as follows: the amygdala is an important component of the brain's limbic system, located in the medial temporal lobe, and plays a key role in emotion processing, memory formation, and social behavior; by extracting keywords, it can be simplified to: limbic system, medial temporal lobe, emotion processing, memory formation, and social behavior. These keywords are then input into the text encoder for embedding generation. The text encoder used in the present invention is the BioBERT model that has been fine-tuned with medical corpus to enhance the representation capability under medical semantics. For any pair of brain regions, the cosine similarity between their semantic embedding vectors is calculated to obtain the semantic similarity value between the two, and it is used as an important basis for edge connection in the brain network diagram. The semantic similarity is calculated by cosine similarity, and the expression is as follows:

[0049]

[0050] Among them, s u and s v denotes the semantic embedding vectors of brain regions u and v, respectively, · denotes the vector dot product operation, and |·| denotes the L2 norm of the vector. The resulting semantic similarity matrix provides the basis for semantic edge connections for the subsequent construction of semantically enhanced brain network graphs.

[0051] S2: Semantically enhanced brain network graph construction module: semantic similarity is used as side information, and functional connectivity values between brain regions are introduced as node features to construct a brain network graph that integrates functional connectivity and semantic relationships. The topological structure of this graph is closer to the potential connection between cognitive and physiological semantic levels. The functional connectivity is generated by calculating the correlation of BOLD signals between each brain region using the Pearson correlation coefficient. The BOLD signal is extracted from the preprocessed fMRI signal based on the brain region template. In this embodiment, the functional connectivity is generated using the BOLD signal divided based on the AAL template in the public data of Rest-meta-MDD.

[0052] S3: Graph neural network training under the node redistribution mechanism: The semantically enhanced brain network graph constructed above is input into the dynamic graph neural network. The dynamic graph neural network includes two layers of graph neural networks, and a node redistribution mechanism is added between the two layers of graph neural networks. The input of the first layer of graph neural network is the semantically enhanced brain network graph, and its output corresponds to the input of the node redistribution mechanism. The input of the second layer of graph neural network is the brain network graph simplified by the node redistribution mechanism. The node redistribution mechanism dynamically reconstructs the graph structure according to the feature representation of the nodes, improves the expressive power of the graph representation and removes redundant connections. Assume that the node feature matrix in the graph is Where N represents the number of nodes and D represents the feature dimension of each node. Each row of the node feature matrix represents the feature representation of node i.

[0053] In order to achieve adaptive aggregation and simplification at the structural level based on node features, a feature-to-group mapping is designed. This mapping introduces a learnable weight matrix It is used to map the original features to the embedding space of G shared groups. Then, the group assignment probability of each node is normalized through the Softmax operation to obtain the group assignment matrix The expression is as follows:

[0054] M=Softmax(XW)

[0055] Each row in the group assignment matrix M represents the normalized assignment probability of a node among G groups.

[0056] To achieve structural reconstruction, for each node, the final group to which it belongs is determined based on the group with the highest probability of being assigned. j ,Right now:

[0057]

[0058] Among them, M ij is the mapping probability of node i to group j in the group assignment matrix. The non-differentiability of the argmax operation hinders end-to-end optimization of the model during training. Therefore, the present invention introduces the Gumbel-Softmax mechanism as a differentiable alternative. Gumbel-Softmax can simulate sampling from a discrete distribution by introducing Gumbel noise and uses a temperature parameter τ to control the smoothness of the sampling. Specifically, the group assignment probability of node i is:

[0059]

[0060] where g ij Is to use Gumbel-Softmax to M ij The noise after resampling, exp is the exponential function, and log is the logarithmic function.

[0061] Therefore, group belonging

[0062] This mechanism maintains classification discriminability while ensuring the feasibility of gradient propagation, supporting end-to-end model training. After the nodes are divided into groups, the original graph is restructured according to the group assignments: nodes belonging to the same group are aggregated to construct a simplified graph structure. In this simplified graph, each group is treated as a new node, and the connections between groups are determined based on the original connections between nodes within the group.

[0063] The node redistribution mechanism realizes the dynamic reconstruction of the graph structure from the node level to the group level, retains high-quality connections, and compresses redundant edges, thereby improving the learning efficiency and generalization ability of graph neural networks. It is particularly suitable for modeling large-scale brain network graphs and improving feature extraction and prediction performance in downstream tasks.

[0064] S4: Graph feature extraction and prediction module; after completing node feature learning and structure reconstruction, the updated graph structure and its corresponding node feature matrix are obtained In order to extract the representation information of the graph layer, the present invention adopts a dual-channel graph-level pooling strategy, that is, a combination of average pooling and maximum pooling:

[0065]

[0066] Where x′ i ∈X′ is the feature vector of each node. The two aggregation results are concatenated to obtain the final graph-level representation:

[0067]

[0068] Where || represents vector concatenation. This vector combines the global average features and the most significant local features of the node information in the entire graph, and has stronger discrimination ability. For the i-th sample graph-level representation h graph,i Input the multi-layer perceptron classifier to get the final output prediction And use classification loss to optimize the network. The expression is as follows:

[0069]

[0070] Where N2 represents the number of samples, y i Represents the true label of the input i-th sample.

[0071] S5: By minimizing the classification loss function, the backpropagation algorithm is used to continuously update the learnable parameters in the neural network. Training is terminated when the model performance reaches a preset accuracy threshold or the number of training rounds reaches a set upper limit. The resulting optimal model can be used to assist in the analysis of MDD data. By inputting new brain network feature data, including the fMRI signal to be predicted and predefined brain region templates, classification results for brain fMRI signal data can be achieved.

[0072] Experiments were conducted on the public data of Rest-meta-MDD, comparing the proposed graph neural network model based on semantic modeling and node reallocation to a traditional graph neural network without semantic modeling and node reallocation. The graph neural network model based on semantic modeling and node reallocation outperformed the traditional graph neural network, improving classification accuracy by 2.19%. While improving accuracy, semantic modeling enriches the representation of brain networks, while the node reallocation mechanism effectively simplifies the graph structure and reduces computational complexity.

[0073] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A prediction model construction method based on semantic modeling and dynamic graph neural network, characterized in that: The following steps are involved: Step 1: Semantic representation of brain region nodes. A large language model is used to semantically model the anatomical structure and functional attributes of each brain region based on domain knowledge. The large language model is then used to extract representative terms or keywords from the brain region descriptions as the core concept semantic representations of the brain region. The core concept semantic representations are input into a text encoder to obtain semantic embedding vectors, and the similarity between the semantic embedding vectors is then calculated. Step 2: Construction of brain network graph based on semantic enhancement; Acquire functional magnetic resonance imaging (fMRI) signal datasets and calculate functional connectivity values between brain regions. Use semantic similarity as side information and functional connectivity values between brain regions as node features to construct a brain network diagram. Step 3: Construct a dynamic graph neural network. The dynamic graph neural network includes two layers of graph neural networks. A node reallocation mechanism is added between the two layers of graph neural networks. The node reallocation mechanism dynamically reconstructs the graph structure according to the node feature representation; the node features are updated through the dynamic graph neural network. Step 4: Graph feature extraction and prediction: Aggregate the updated node features based on a dual-channel graph-level pooling strategy to obtain a graph-level representation, which is then input into a multi-layer perceptron classifier to obtain the final prediction result. Step 5: Optimize the model using classification loss to obtain a trained prediction model.

2. The prediction model construction method based on semantic modeling and dynamic graph neural network according to claim 1 is characterized in that: The dynamic graph neural network is specifically as follows: The input of the first-layer graph neural network is the semantically enhanced brain network graph, and its output corresponds to the input of the node reallocation mechanism. The input of the second-layer graph neural network is the brain network graph simplified by the node reallocation mechanism. The node reallocation mechanism is as follows: Suppose the node feature matrix in the graph output by the first graph neural network is Where N represents the number of nodes, and D represents the feature dimension of each node; Based on the mapping of features to groups, the group assignment matrix is obtained The expression is as follows: M=Softmax(XW) Among them, each row in the group assignment matrix M represents the normalized assignment probability of a node among G groups. is a learnable weight matrix; For each node i, the group with the highest probability of being assigned is determined to be the final group. j , the expression is as follows: Among them, M ij is the mapping probability of node i to group j in the group assignment matrix, The actual group assignment probability for node i, g ij Is to use Gumbel-Softmax to M ij The noise after resampling, exp is the exponential function, log is the logarithmic function, and τ is the temperature parameter; After completing the group division of the nodes, the original graph is restructured according to the group assignment results: the nodes belonging to the same group are aggregated to construct a simplified graph structure. In this simplified graph structure, each group is regarded as a new node, and the connection relationship between groups is determined based on the original connection between nodes in the group.

3. The prediction model construction method based on semantic modeling and dynamic graph neural network according to claim 2 is characterized in that: The text encoder is a BioBERT model fine-tuned with medical corpus.

4. The method for constructing a prediction model based on semantic modeling and dynamic graph neural network according to claim 3, characterized in that: The similarity is calculated by cosine similarity, which is expressed as follows: Among them, s u and s v denote the semantic embedding vectors of brain region u and brain region v respectively, . denotes the vector dot product operation, and |·| denotes the L2 norm of the vector.

5. The method for constructing a prediction model based on semantic modeling and dynamic graph neural network according to claim 4, characterized in that: The functional connectivity values between the brain regions are generated by calculating the correlation of BOLD signals between the brain regions using the Pearson correlation coefficient, and the BOLD signals are extracted from the preprocessed fMRI signals according to the brain region template.

6. The method for constructing a prediction model based on semantic modeling and dynamic graph neural network according to claim 5, characterized in that: The dual-channel graph-level pooling strategy is as follows: Where x′ i is the updated node feature, h avg represents the average pooling aggregation result, h max Represents the maximum pooling aggregation result; the two aggregation results are spliced together to obtain the final graph-level representation: Where || represents vector concatenation.

7. A prediction device based on semantic modeling and dynamic graph neural network, characterized in that: include: Brain Region Node Semantic Representation Module: This module uses a large language model to semantically model the anatomical structure and functional attributes of each brain region based on domain knowledge. It then uses the large language model to extract representative terms or keywords from the brain region descriptions as the core concept semantic representations of the brain region. The core concept semantic representations are input into a text encoder to obtain semantic embedding vectors, and the similarity between the semantic embedding vectors is then calculated. Semantic-enhanced brain network graph construction module: acquires functional magnetic resonance imaging (fMRI) signal data and calculates the functional connectivity values between brain regions; uses semantic similarity as side information and the functional connectivity values between brain regions as node features to construct a brain network graph; Node reallocation graph neural network learning module: This module includes two layers of graph neural networks, between which a node reallocation mechanism is added. The node reallocation mechanism dynamically reconstructs the graph structure based on the node feature representation; node features are updated through the dynamic graph neural network. Graph feature extraction and prediction module; based on the dual-channel graph-level pooling strategy, the updated node features are aggregated to obtain a graph-level representation, which is then input into the multi-layer perceptron classifier to obtain the final prediction result.

Images