Method and device for constructing a prediction model based on semantic modeling and dynamic graph neural networks

Abstract

The application provides a prediction model construction method and device based on semantic modeling and a dynamic graph neural network, and relates to the field of image processing. The application aims at the rich information representation contained in the brain region template, constructs a unified and structured brain region semantic description system through LLM assistance, provides a new paradigm for brain network semantic modeling, and enhances the knowledge interpretability of the brain atlas. The semantic enhanced brain network graph significantly improves the richness and expression ability of the brain network representation by fusing the semantic information of the brain region structure and function. The node redistribution mechanism is introduced on the basis of the traditional GNN framework, which can adaptively reconstruct the graph structure according to the node characteristics, effectively remove redundant connections, highlight key areas, and improve the generalization ability and training efficiency of the model. The model provided by the application combines semantic modeling and functional connection, can more accurately capture the abnormal pattern of the brain network, and provides a more efficient and accurate intelligent means for the auxiliary classification of brain region detection data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a prediction model construction method and device based on semantic modeling and dynamic graph neural networks. BACKGROUND

[0002] Major Depressive Disorder (MDD) as the second most disabling mental illness in the world, its diagnosis has long relied on the subjective evaluation of the symptom checklist of the Diagnostic and Statistical Manual of Mental Disorders (DSM), lacking objective biomarker support. In existing clinical practice, doctors need to make judgments by combining patient self-reports, behavioral observations, and exclusion tests, but this model is easily disturbed by individual differences and comorbid factors, with a misdiagnosis rate as high as 30-40%.

[0003] Functional Magnetic Resonance Imaging (fMRI) detects Blood-Oxygen-Level Dependent (BOLD) signals to indirectly reflect the timing changes of neural activity. BOLD signals, as an important measure of brain activity, are the core basis for constructing functional connectivity networks (FC). Meanwhile, in the auxiliary judgment of complex mental disorders such as MDD, there are dynamic abnormal changes in the brain functional connectivity of patients, so FC is often used as an important feature input for computer-aided judgment. In constructing FC networks, the whole brain BOLD signal is usually spatially divided based on a pre-defined brain region template, and the divided brain regions are used as network nodes to determine the connection edge weights by calculating their correlation. However, traditional methods often treat each brain region as a single, static node, ignoring its potential multi-dimensional information related to structure, function, and disease, which limits the expression ability of brain networks and affects the discrimination effect of major depressive disorder. At the same time, different brain region templates have different division standards, and lack of systematic and structured brain region function description texts, which further limits the deep modeling of brain region semantic features.

[0004] With the widespread application of Large Language Models (LLM) in the field of medical health, its powerful language understanding and knowledge extraction capabilities provide a possible way for brain region semantic modeling. By generating function description corpus related to brain regions through LLM, combining key concept extraction and semantic feature encoding methods, more rich context information can be introduced for each node, thereby improving the expression ability of brain network modeling. However, existing work still has gaps in how to integrate semantic information with graph brain networks.

[0005] Meanwhile, as an important progress in the field of graph structure modeling in recent years, Graph Neural Network (GNN) has been widely used in brain network analysis tasks. GNN effectively captures the connection relationship between brain regions and the interaction of node features through the information aggregation mechanism of adjacency structure, has good feature extraction capability, and is especially suitable for the fusion modeling of complex brain graph structure and multi-modal features. However, existing researches mainly focus on static node brain networks, and there is still a lack of unified framework and effective strategies for how to effectively embed semantic enhanced brain region node information and dynamically learn the graph structure to realize the feature recognition of major depressive disorder. This limits the utilization of semantic information of brain region nodes in the process of brain network construction in the graph neural network, and the graph structure cannot be dynamically updated according to individual characteristics, which is difficult to meet the demand of precise diagnosis of MDD for high expression and high interpretability model. SUMMARY

[0006] The present application aims to utilize large language models and graph neural networks to provide a graph neural network model based on semantic modeling and node reassignment, which enriches the modeling information of brain network graph by combining fMRI data and brain region template features, and improves the classification accuracy of fMRI signal data by dynamically reconstructing the graph structure in the learning process of graph neural network using node reassignment mechanism.

[0007] The present application utilizes large language models for semantic modeling and adopts node reassignment mechanism to reconstruct brain network graph in the training process of GNN, thereby realizing efficient feature learning of brain network. The graph neural network based on semantic modeling and node reassignment described in the present application includes a brain region node semantic representation module, a semantic enhanced brain network graph, a node reassignment graph neural network learning module, and a graph feature extraction and prediction module. The brain region node semantic representation module utilizes LLM to extract and model the structural and functional information of brain regions, generates semantic descriptions of brain regions with unified standards, and embeds the description information through a text encoder to obtain the semantic similarity between brain regions. The semantic enhanced brain network graph generates semantic edges between brain regions based on the above semantic similarity, and constructs a brain network that integrates semantic information by taking the functional connection strength of each brain region with other brain regions as node features. The node reassignment graph neural network learning module introduces a node reassignment mechanism based on the traditional GNN framework, dynamically adjusts the connection structure of the graph according to the feature similarity between nodes, thereby compressing redundant connections, highlighting key information and improving the expression ability of the graph structure. The graph feature extraction and prediction module fuses the node features in the graph through average pooling and max pooling strategies, and then completes the classification and prediction tasks of brain function magnetic resonance imaging data through a multi-layer perceptron.

[0008] The prediction model construction method based on semantic modeling and dynamic graph neural network comprises the following steps:

[0009] Step 1: semantic representation of brain region nodes; the semantic modeling of the anatomical structure and functional attributes of each brain region is performed based on the domain knowledge of a large language model, and then representative terms or keywords are extracted from the brain region description by the large language model to serve as the core concept semantic representation of the brain region; the core concept semantic representation is input into a text encoder to obtain semantic embedding vectors, and then the similarity between the semantic embedding vectors is calculated;

[0010] Step 2: brain network graph construction based on semantic enhancement; a functional magnetic resonance imaging (fMRI) signal dataset is obtained, and the functional connection values between brain regions are calculated; the semantic similarity is taken as edge information, and the functional connection values between brain regions are taken as node features to construct a brain network graph; the functional connection values between brain regions are generated by calculating the correlation of BOLD signals between brain regions by Pearson correlation coefficient, and the BOLD signals are extracted from the preprocessed fMRI signals according to the brain region template.

[0011] Step 3: constructing a dynamic graph neural network, which comprises two layers of graph neural networks, and a node reassignment mechanism is added between the two layers of graph neural networks, which dynamically reconstructs the graph structure according to the node features; the node features are updated by the dynamic graph neural network;

[0012] Step 4: graph feature extraction and prediction; the updated node features are aggregated based on a double-channel graph-level pooling strategy to obtain a graph-level representation, which is input into a multilayer perceptron classifier to obtain the final prediction result;

[0013] Step 5: optimizing the model with a classification loss to obtain a trained prediction model.

[0014] The dynamic graph neural network is as follows:

[0015] The input of the first layer of graph neural network is the semantic enhanced brain network graph, and the output thereof corresponds to the input of the node reassignment mechanism, and the input of the second layer of graph neural network is the brain network graph simplified by the node reassignment mechanism;

[0016] The node reassignment mechanism is as follows:

[0017] Let the node feature matrix of the graph output by the first graph neural network be Where N represents the number of nodes, and D represents the feature dimension of each node.

[0018] Based on the feature-to-group mapping, a group assignment matrix is obtained The expression is as follows:

[0019] M = Softmax(XW)

[0020] where each row of the group assignment matrix M represents the normalized assignment probability of a node to G groups, is a learnable weight matrix;

[0021] For each node i, the group with the highest assignment probability is determined as the final group affiliation group j , the expression is as follows:

[0022]

[0023] where M ij is the mapping probability of node i to group j in the group assignment matrix, is the actual group assignment probability of node i, g ij is the noise after re-sampling M ij using Gumbel-Softmax, exp is the exponential function, log is the logarithmic function, and tau is the temperature parameter;

[0024] After completing the group division of the nodes, the original graph is reconstructed according to the group assignment result: the nodes belonging to the same group are aggregated to construct a simplified graph structure, in which each group is regarded as a new node, and the connection relationship between groups is determined according to the original connection between nodes in the group.

[0025] The application also provides a prediction device based on semantic modeling and dynamic graph neural network, comprising:

[0026] The brain region node semantic representation module: a large language model is used to model the anatomical structure and functional attributes of each brain region based on domain knowledge, and then representative terms or keywords are extracted from the brain region description by using the large language model, which are used as the core concept semantic representation of the brain region; the core concept semantic representation is input into a text encoder to obtain a semantic embedding vector, and then the similarity between the semantic embedding vectors is calculated.

[0027] The semantic-enhanced brain network graph construction module: obtain functional magnetic resonance imaging (fMRI) signal data, calculate the functional connection values between brain regions; the semantic similarity is used as edge information, and the functional connection values between brain regions are used as node features to construct a brain network graph.

[0028] The node reassignment graph neural network learning module: including two layers of graph neural networks, a node reassignment mechanism is added between the two layers of graph neural networks, which dynamically reconstructs the graph structure according to the feature representation of the node; the node features are updated through the dynamic graph neural network.

[0029] The updated node features are aggregated based on a double-channel graph-level pooling strategy to obtain a graph-level representation, which is input into a multilayer perceptron classifier to obtain a final prediction result.

[0030] The beneficial effects of the present application are as follows:

[0031] A unified and structured semantic description system of brain regions is constructed with the aid of LLM, providing a new paradigm for brain network semantic modeling and enhancing the knowledge interpretability of brain atlases.

[0032] The semantic information of brain region structure and function significantly improves the richness and expressiveness of brain network representation.

[0033] The node reassignment mechanism can adaptively reconstruct the graph structure according to the node features, effectively removing redundant connections and highlighting key areas to improve the generalization ability and training efficiency of the model.

[0034] The proposed model combines semantic modeling and functional connections to more accurately capture abnormal patterns of brain networks, providing more efficient and accurate intelligent means for data processing and analysis of brain function magnetic resonance imaging of mental illnesses. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 It is a model construction framework of graph neural network based on semantic modeling and node reassignment.

[0036] Figure 2 It is the generation process and instance of semantic representation of brain region nodes.

[0037] Figure 3 It is the training and prediction process of graph neural network based on semantic modeling and node reassignment. DETAILED DESCRIPTION

[0038] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and examples, so that those skilled in the art can better understand the present application.

[0039] As shown in Figure 1 The present embodiment provides a prediction device based on semantic modeling and dynamic graph neural network, which comprises:

[0040] The brain region node semantic representation module: based on the field knowledge, the anatomical structure and functional attributes of each brain region are semantically modeled by a large language model, and then representative terms or keywords are extracted from the brain region description by the large language model as the core concept semantic representation of the brain region; the core concept semantic representation is input into a text encoder to obtain a semantic embedding vector, and then the similarity between the semantic embedding vectors is calculated.

[0041] A semantic-enhanced brain network graph construction module: obtain functional magnetic resonance imaging (fMRI) signal data, calculate the functional connection value between brain regions; take the semantic similarity as the edge information and the functional connection value between brain regions as the node feature to construct a brain network graph;

[0042] A graph neural network learning module with node reassignment: including two layers of graph neural networks, a node reassignment mechanism is added between the two layers of graph neural networks, the node reassignment mechanism dynamically reconstructs the graph structure according to the feature representation of the nodes; the node features are updated through the dynamic graph neural network;

[0043] A graph feature extraction and prediction module: based on a double-channel graph-level pooling strategy, the updated node features are aggregated to obtain a graph-level representation, which is input into a multilayer perceptron classifier to obtain a final prediction result.

[0044] Next, the construction method of the prediction model based on semantic modeling and dynamic graph neural network will be further described in combination with the device.

[0045] As shown in Figure 3 , in the embodiment of the present application, the construction method of the prediction model based on semantic modeling and dynamic graph neural network is as follows:

[0046] S1: Brain region node semantic representation module: as shown in Figure 2 , the process of using a large language model (LLM) to model the semantic of the anatomical structure and functional attributes of each brain region based on domain knowledge includes two main steps: brain region description generation and brain region key concept extraction.

[0047] In the brain region description generation stage, the name data in the brain region template is obtained, and in this embodiment, the Anatomical Automatic Labeling (AAL) template is used. By designing a standardized Prompt instruction, the LLM output is limited to contain the structure and function of the brain region description, thereby generating a unified and standardized brain region semantic text.

[0048] In the brain region key concept extraction stage, the LLM is further utilized to extract the top 10 representative terms or keywords from the brain region description generated above as the core concept semantic representation of the brain region. For example, the structure and function description content about the brain region "amygdala" is: the amygdala is an important part of the limbic system, located in the medial temporal lobe, and plays a key role in emotion processing, memory formation, and social behavior; through the extraction of keywords, it can be simplified as: limbic system, medial temporal lobe, emotion processing, memory formation, and social behavior. These keywords are then input into the text encoder for embedding generation. In this invention, the text encoder used is the BioBERT model fine-tuned on medical corpus to improve the representation ability under medical semantics. For any brain region pair, the semantic similarity value between them is obtained by calculating the cosine similarity between their semantic embedding vectors, and it is used as an important basis for edge connection in the brain network graph. The semantic similarity is calculated by the cosine similarity, and the expression is as follows:

[0049]

[0050] where s u and s v represent the semantic embedding vectors of brain region u and brain region v, respectively, · represents the vector dot product operation, and |·| represents the L2 norm of the vector. The final semantic similarity matrix provides a semantic edge connection basis for subsequent construction of a semantic enhanced brain network graph.

[0051] S2: Semantic enhanced brain network graph construction module: The semantic similarity is used as edge information, and the functional connection value between brain regions is introduced as node feature to construct a brain network graph that integrates functional connection and semantic relationship. This graph is more close to the potential relationship at the cognitive and physiological semantic level in terms of topology. The functional connection is generated by calculating the Pearson correlation coefficient of the BOLD signals between brain regions, and the BOLD signals are extracted from the preprocessed fMRI signals according to the brain region template. In this embodiment, the BOLD signals based on the AAL template in the public data of Rest-meta-MDD are used to generate the functional connection.

[0052] S3: Graph neural network training under node reassignment mechanism: The semantic enhanced brain network graph constructed above is input into the dynamic graph neural network, which includes two layers of graph neural networks with a node reassignment mechanism between them. The input of the first layer of graph neural network is the semantic enhanced brain network graph, and its output corresponds to the input of the node reassignment mechanism. The input of the second layer of graph neural network is the brain network graph simplified by the node reassignment mechanism. The node reassignment mechanism dynamically reconstructs the graph structure according to the node features, improves the representation ability of the graph, and removes redundant connections. Let the node feature matrix of the graph be where N denotes the number of nodes and D denotes the feature dimension of each node. Each row of the node feature matrix X represents the feature representation of node i.

[0053] To achieve adaptive aggregation and simplification at the structure level based on node features, a feature-to-group mapping is designed. The mapping introduces a learnable weight matrix for mapping the original features to the embedding space of G shared groups. Subsequently, the probability of group assignment for each node is normalized by the Softmax operation, resulting in a group assignment matrix The expression is as follows:

[0054] M = Softmax(XW)

[0055] Each row of the group assignment matrix M represents the normalized assignment probability of a node among the G groups.

[0056] To achieve structure reconstruction, for each node, the final group affiliation group j is determined according to the group with the highest assignment probability.

[0057]

[0058] where M ij is the mapping probability of node i to group j in the group assignment matrix. Since the argmax operation is not differentiable, it hinders the end-to-end optimization of the model during training. Therefore, the Gumbel-Softmax mechanism is introduced as a differentiable alternative. Gumbel-Softmax can simulate sampling from a discrete distribution by introducing Gumbel noise, and use the temperature parameter τ to control the smoothness of sampling. Specifically, the group assignment probability of node i is:

[0059]

[0060] where g ij is the noise after resampling M ij using Gumbel-Softmax, exp is the exponential function, and log is the logarithmic function.

[0061] Therefore, the group affiliation

[0062] This mechanism maintains the discriminative power of classification while ensuring the feasibility of gradient propagation, supporting end-to-end training of the model. After completing the group division of nodes, the original graph is reconstructed according to the group assignment results: nodes belonging to the same group are aggregated to construct a simplified graph structure. In this simplified graph, each group is considered as a new node, and the connection relationship between groups is determined according to the original connection between nodes within the group.

[0063] The node reassignment mechanism realizes dynamic reconstruction of the graph structure from the node level to the group level, preserves high-quality connections, and compresses redundant edges, thereby improving the learning efficiency and generalization ability of the graph neural network, and is especially suitable for modeling large-scale brain network graphs and improving feature extraction and prediction performance in downstream tasks.

[0064] S4: graph feature extraction and prediction module; after completing node feature learning and structure reconstruction, an updated graph structure and its corresponding node feature matrix are obtained In order to extract the representation information at the graph level, the present application adopts a double-channel graph-level pooling strategy, that is, the combination of average pooling (Average Pooling) and maximum pooling (Max Pooling):

[0065]

[0066] wherein x' i is the feature vector of each node. The two aggregation results are spliced to obtain the final graph-level representation:

[0067]

[0068] wherein || represents vector splicing. The vector combines the global average feature and the local most significant feature of the node information in the whole graph, and has stronger discriminative ability. For the i-th sample graph-level representation h graph,i The final output prediction is obtained by inputting a multilayer perceptron classifier and the network is optimized by using a classification loss. The classification loss The expression is as follows:

[0069]

[0070] wherein N2 represents the number of samples, y i represents the true label of the i-th sample.

[0071] S5: by minimizing the classification loss function, the learning parameters in the neural network are continuously updated by using a back propagation algorithm. When the model performance reaches a preset accuracy threshold or the training round number reaches a set upper limit, the training process is terminated. The finally obtained optimal model can be applied to auxiliary data analysis of MDD, and by inputting new brain network feature data, including to-be-predicted fMRI signals and a predefined brain region template, the classification result of the brain fMRI signal data is realized.

[0072] Experiments are conducted on the public data of Rest-meta-MDD for the graph neural network model based on the semantic modeling and node reassignment mechanism and the traditional graph neural network without the semantic modeling and node reassignment mechanism, and the graph neural network model based on the semantic modeling and node reassignment mechanism is better than the traditional graph neural network, and the classification accuracy is improved by 2.19%. While improving the accuracy, the semantic modeling enriches the representation of the brain network, and the node reassignment mechanism can effectively simplify the graph structure and reduce the computational complexity.

[0073] It can be understood that the present application is described by some embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, these features and embodiments can be modified to adapt to specific conditions and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the scope of the present application.

Claims

1. A method for constructing a prediction model based on semantic modeling and dynamic graph neural network, characterized in that, The method comprises the following steps: Step 1: semantic representation of brain region nodes; using a large language model to semantically model the anatomical structure and functional attributes of each brain region based on domain knowledge, and then using the large language model to extract representative terms or keywords from the brain region description as the core concept semantic representation of the brain region; inputting the core concept semantic representation into a text encoder to obtain semantic embedding vectors, and then calculating the similarity between the semantic embedding vectors; Step 2: construction of a brain network graph based on semantic enhancement; Obtain a functional magnetic resonance imaging (fMRI) signal dataset, calculate the functional connection values between brain regions, and construct a brain network graph by taking the semantic similarity as edge information and the functional connection values between brain regions as node features; Step 3: constructing a dynamic graph neural network, which comprises two layers of graph neural networks, and adding a node redistribution mechanism between the two layers of graph neural networks, which dynamically reconstructs the graph structure according to the node features; updating the node features through the dynamic graph neural network; Step 4: graph feature extraction and prediction; aggregating the updated node features based on a double-channel graph-level pooling strategy to obtain a graph-level representation, which is input into a multilayer perceptron classifier to obtain the final prediction result; Step 5: optimizing the model with a classification loss to obtain a trained prediction model.

2. The prediction model construction method based on semantic modeling and dynamic graph neural network according to claim 1, characterized in that, The dynamic graph neural network is as follows: The input of the first layer of graph neural networks is the semantic enhanced brain network graph, and the output corresponds to the input of the node redistribution mechanism; the input of the second layer of graph neural networks is the simplified brain network graph of the node redistribution mechanism; The node redistribution mechanism is as follows: Let the node feature matrix in the graph output by the first graph neural network be where N represents the number of nodes, and D represents the feature dimension of each node. Based on the mapping of features to groups, a group assignment matrix is obtained The expression is as follows: M = Softmax(XW) wherein each row of the group allocation matrix M represents a normalized allocation probability of a node among the G groups, is a learnable weight matrix; For each node i, determine the final group affiliation group to which it is assigned with the highest probability j The expression is as follows: where M ij is the mapping probability from node i to group j in the component assignment matrix, is the actual group assignment probability for node i, g ij is the noise after re-sampling M ij using Gumbel-Softmax, exp is the exponential function, log is the logarithmic function, and τ is the temperature parameter; After completing the group division of the nodes, the original graph is reconstructed according to the group allocation results: the nodes belonging to the same group are aggregated to construct a simplified graph structure, in which each group is regarded as a new node, and the connection relationship between groups is determined according to the original connection between the nodes in the group.

3. The method of claim 2, wherein the method further comprises: The text encoder is a BioBERT model fine-tuned on medical corpus.

4. The method of claim 3, wherein, The similarity is calculated by cosine similarity, and the expression is as follows: where s u and s v denote semantic embedding vectors for brain region u and brain region v, respectively, • denotes the vector dot product operation, and | · | denotes the L2 norm of a vector.

5. The method of claim 4, wherein, The functional connection values between brain regions are generated by calculating the correlation of BOLD signals between brain regions using Pearson correlation coefficient, and the BOLD signals are extracted from the preprocessed fMRI signals according to the brain region template.

6. The method of claim 5, wherein the method further comprises: The double-channel graph-level pooling strategy is as follows: where x′ i is the updated node feature, h avg denotes the average pooling aggregation result, h max denotes the max pooling aggregation result; and the two aggregation results are spliced to obtain the final graph-level representation: Where || represents vector splicing.

7. A prediction device based on semantic modeling and dynamic graph neural networks, characterized in that, It comprises: a brain region node semantic representation module: using a large language model to semantically model the anatomical structure and functional attributes of each brain region based on domain knowledge, and then using the large language model to extract representative terms or keywords from the brain region description as the core concept semantic representation of the brain region; inputting the core concept semantic representation into a text encoder to obtain semantic embedding vectors, and then calculating the similarity between the semantic embedding vectors; a semantic enhanced brain network graph construction module: obtaining a functional magnetic resonance imaging (fMRI) signal dataset, calculating the functional connection values between brain regions, and constructing a brain network graph by taking the semantic similarity as edge information and the functional connection values between brain regions as node features; The node reassignment graph neural network learning module comprises two layers of graph neural networks, and a node reassignment mechanism is added between the two layers of graph neural networks, wherein the node reassignment mechanism dynamically reconstructs a graph structure according to a feature representation of a node; and the node feature is updated through the dynamic graph neural network; The graph feature extraction and prediction module aggregates the updated node features based on a double-channel graph-level pooling strategy to obtain a graph-level representation, and inputs the graph-level representation into a multilayer perceptron classifier to obtain a final prediction result.

Images