Clinical traditional Chinese medicine knowledge graph construction method based on multi-modal graph attention network

Abstract

The invention provides a clinical traditional Chinese medicine knowledge graph construction method based on a multi-modal graph attention network. The method comprises the following steps: acquiring multi-modal data in a traditional Chinese medicine clinical environment; generating a corresponding type label; all modal vectors in the multi-modal semantic representation set are corresponding to nodes, and an initial traditional Chinese medicine clinical knowledge graph is constructed; based on attention enhancement of a graph neural mechanism, generating a graph after structure enhancement; obtaining a standard role set of the traditional Chinese medicine knowledge base, performing semantic role recognition, and generating a node set with role tags; calculating an edge importance score according to the normalized map, and screening out a final edge structure based on the edge importance score to generate a standardized map structure; and importing the standardized map structure into a map database to realize clinical knowledge calling. According to the method, a complete closed-loop process from original clinical data to deployment output is realized through multi-modal modeling, structure generation, graph optimization, role modeling and semantic feedback self-evolution.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-modal graph attention network, and particularly relates to a clinical traditional Chinese medicine knowledge graph construction method based on a multi-modal graph attention network. BACKGROUND

[0002] Traditional Chinese medicine is an important part of traditional Chinese medicine, and in clinical diagnosis and treatment, it emphasizes syndrome differentiation and treatment, multiple symptoms and compound compatibility. Its knowledge structure has the characteristics of strong semantic dependence, high non-structured and modal diversity. In modern clinical practice of traditional Chinese medicine, the data handled by doctors is no longer limited to structured text, but also includes multi-modal information such as tongue image, audio of inquiry and video observation. There are significant semantic differences and inconsistent forms among these data, which seriously restricts the applicability of traditional knowledge graph construction methods. Current most of the construction schemes of traditional Chinese medicine knowledge graph still rely on ancient books or pharmacopoeia data, and cannot automatically process clinical data. They lack the processing ability of image and voice modalities. The structure of the graph is static, the semantics deviates from the logic of traditional Chinese medicine, and it is difficult to dynamically adjust with the change of clinical knowledge, resulting in serious lack of practicality and scalability.

[0003] With the development of modern artificial intelligence technology, although multi-modal learning and graph neural network have shown potential in graph modeling, their application in the field of traditional Chinese medicine is still in the exploratory stage. The main problems of current technology include: unable to effectively interface with multi-modal data of traditional Chinese medicine and realize unified semantic representation, unable to fully embed the knowledge structure of traditional Chinese medicine to realize semantic constraint graph building, and lack of ability to optimize the structure of the graph driven by semantic feedback. Therefore, a complete solution is needed to integrate multi-modal feature expression, combine with the semantic rules of traditional Chinese medicine ontology, support the continuous evolution of the structure, and have the output ability of service, so as to truly land the construction of intelligent knowledge graph of traditional Chinese medicine for clinical practice. SUMMARY

[0004] The purpose of the present application is to solve the key problems of multi-modal heterogeneous input processing difficulty, inconsistency between graph structure and traditional Chinese medicine knowledge logic, and lack of self-evolution ability of the graph in the actual clinical application of traditional Chinese medicine graph.

[0005] In order to achieve the above purpose, the present application provides a clinical traditional Chinese medicine knowledge graph construction method based on multi-modal graph attention network, which comprises:

[0006] Obtaining multi-modal data in the clinical environment of traditional Chinese medicine, including text data, image data and audio data, and mapping the multi-modal data into a multi-modal semantic representation set and a modal label set corresponding to the modal vector, respectively; wherein the modal label set is used to record the modal source of each modal vector;

[0007] Obtain a traditional Chinese medicine knowledge graph ontology and a standard type, make attribution judgment on all modal vectors in the multi-modal semantic representation set according to the standard type, to obtain corresponding type labels; correspond all modal vectors in the multi-modal semantic representation set to nodes to construct an initial traditional Chinese medicine clinical knowledge graph G0; wherein the edges of the initial traditional Chinese medicine clinical knowledge graph are obtained through connection score calculation on the nodes;

[0008] Perform attention enhancement based on a graph neural mechanism on the initial traditional Chinese medicine clinical knowledge graph G0 to generate a structure-enhanced graph G1;

[0009] Obtain a standard role set of a traditional Chinese medicine knowledge base, perform semantic role recognition on each node in the structure-enhanced graph G1 based on the standard role set to generate a node set with role labels, and update the structure-enhanced graph G1 to obtain a normalized graph G2;

[0010] Calculate edge importance scores according to the normalized graph G2, and filter out final edge structures based on the edge importance scores to generate a standardized graph structure G final ;

[0011] Import the standardized graph structure G final into a graph database, and construct a RESTful interface to realize clinical knowledge calling.

[0012] Further, the step of mapping the multi-modal data into a multi-modal semantic representation set and a corresponding modal label set of modal vectors respectively comprises:

[0013] Each entry in the text data is input into a Chinese encoder of a BERT model fine-tuned based on traditional Chinese medicine corpus to generate a vector representation of the corresponding text;

[0014] Each image of the image data is input into a ResNet-50 network that has undergone transfer learning, and finally output is a visual feature vector of the corresponding image;

[0015] The speech data is first transcribed into text by a Wenet model, and then input into the BERT model fine-tuned based on traditional Chinese medicine corpus to obtain a speech semantic representation of the corresponding text;

[0016] Three groups of modal projection matrices are defined for the text data, image data and audio data respectively, and linear transformation is performed on the three groups of modal projection matrices respectively, finally all modalities are mapped to the same d-dimensional semantic space to obtain the modal label set, represented as:

[0017]

[0018] wherein m e {t, i, a} represents a modality type, represents an original vector from modality m, W m is a projection matrix for the corresponding modality, represents a final unified semantic embedding vector.

[0019] wherein the set of modality labels is represented as M = {m k} and each f k is labeled with its original modality source.

[0020] Further, the standard types include symptoms, syndromes, body constitution types, prescriptions, single herbs, treatment methods, etiology and pathogenesis, meridian sites, tongue appearances, pulse appearances, chief complaint descriptions, auxiliary examination results, treatment responses, health preservation suggestions, and dietary taboos.

[0021] Further, the step of obtaining a TCM knowledge graph ontology and standard types, and performing attribution judgment on all modality vectors in the set of multi-modal semantic representations according to the standard types to obtain corresponding type labels specifically includes:

[0022] performing type determination on each modality vector in the set of modality labels based on the standard types to obtain a predicted type label; wherein each type corresponds to a prototype vector;

[0023] attaching all modality vectors f k to nodes v k to form a node set V0 = {v k}, each node being attached with a triple as a predicted type label;

[0024] performing cosine similarity calculation on all modality vectors f k based on entity connection rules in the TCM ontology to obtain initial connection scores of the modality vectors f k , and screening modality vectors with initial connection scores greater than a preset threshold to form a corresponding edge set E0 = {e ij}, e ij being an edge corresponding to modality i and j;

[0025] based on the node set V0 = {v k} and the edge set E0 = {e ij}, obtaining the initial TCM clinical knowledge graph structure G0 = (V0, E0).

[0026] Further, the step of performing attention enhancement based on a graph neural mechanism on the initial TCM clinical knowledge graph G0 to generate a structure-enhanced graph G1 specifically includes:

[0027] For the initial TCM clinical knowledge graph G0, for each pair of adjacent nodes (v i ,v j Calculate its attention score α ij , is represented as:

[0028]

[0029] in, The linear transformation matrix for node features is used to uniformly adjust the representation space of the original semantic vector; The attention scoring vector; ψ(m) i ,m j ) is the modal compatibility function, using empirical weights to represent the confidence level of modal combinations; S ij The initial connection score; R ij It is a semantic frequency penalty factor, defined as a function of the sum of the frequencies of the concepts corresponding to two nodes in the graph, specifically R. ij =log(1+freq(v) i )+freq(v j ); β, γ, η are three control terms hyperparameters that control modal bias, structural confidence, and frequency suppression strength, respectively; || denotes vector concatenation;

[0030] After the attention score is calculated, the node representation is updated using standard attention-weighted aggregation to obtain the updated node representation.

[0031] Multi-stage propagation is performed based on the updated representation of the nodes to obtain the structurally enhanced graph G1.

[0032] Furthermore, the steps of obtaining a standard role set for the TCM knowledge base, performing semantic role recognition on each node in the structure-enhanced graph G1 based on the standard role set to generate a set of nodes with role labels, and updating the structure-enhanced graph G1 to obtain the normalized graph G2 specifically include:

[0033] The representation of each node in the structure-enhanced graph G1 is input into a pre-constructed two-layer neural network, which outputs the predicted role of the corresponding node; the loss function of the two-layer neural network is... Represented as:

[0034]

[0035] Where CE(·) is the cross-entropy loss, y k For node v k Target role tag; r i ,r j For node v i ,v jPredicted role tags; This is an indicator function; it returns 1 if the roles of two nodes should not logically be connected. ij f′ represents the attention weight of the current edge; λ is the penalty coefficient; k This represents the nodes in the enhanced graph G1; AGG(N) k () represents the average value of the neighboring nodes of the current node;

[0036] Based on the predicted role, the structural edge E1 of the structure-enhanced graph G1 is normalized to generate the normalized edge.

[0037] By combining the predicted roles and the nodes of the structure-enhanced graph G1, a set of nodes with role labels is generated, and by combining the normalized edges, a normalized graph G2 is obtained.

[0038] Furthermore, the step of normalizing the structural edges E1 of the structure-enhanced graph G1 based on the predicted role and generating the normalized edges includes:

[0039] For the edge (v) of the structure-enhanced graph G1 i ,v j If the combination of the role labels of its two end nodes is It is determined to be a semantic mismatch, and its current edge's attention weight α ij If the value is less than a set threshold, it will be removed from the set of structural edges.

[0040] Further, the edge importance score is calculated based on the normalized graph G2, and the final edge structure is selected based on the edge importance score to generate the normalized graph structure G. final The specific steps include:

[0041] The nodes of the normalized graph G2 are measured by cosine similarity and adjusted based on the predicted role. The edge importance score is obtained by combining the graph degree of the current node of the normalized graph G2 with the normalized average value.

[0042] The normalized graph G2 is updated based on the edge importance scores, and the update strategy is as follows:

[0043] If the edge importance score is less than a first preset threshold, the edge is marked as low contribution, and the system adjusts the attention weight of its current edge from α. ij Linear decay to 0.1·α ij ;

[0044] If the importance score of an edge is greater than the first preset threshold and the edge is not yet in the structure, a new edge will be automatically added and the attention weight of the new edge will be 0.3.

[0045] If other cases remain original edges and edge weights;

[0046] Based on the generation of standardized atlas structure G after updating strategy final .

[0047] Further, the standardized atlas structure G final Imported into the graph database, the steps of constructing RESTful interface to realize the clinical knowledge call specifically include:

[0048] The nodes of the standardized atlas structure G final Into a set of execution nodes, each execution node contains a standard field; the edges of the standardized atlas structure G final Into the relationship type of the graph database, and the corresponding edge weights are reserved;

[0049] According to the TCM application function of the atlas application template, a service interface is constructed; wherein each typical clinical task is encapsulated as a RESTful API interface, and the internal logic of the interface depends on the path query and edge weight accumulation algorithm of the graph database; wherein the atlas application template is used to define the calling function in the deployment scene; the RESTful API interface supports view calling and modular updating according to the mode.

[0050] Further, the structure of the double-layer neural network is:

[0051] The first layer is a 128-dimensional fully connected layer using ReLU activation, and the second layer is a Softmax layer.

[0052] The beneficial technical effects of the present application are at least the following points:

[0053] This method first maps TCM clinical information from three modalities—text, image, and audio—to the same embedding space through a unified semantic representation mechanism, and introduces modal alignment projection and modal attention weighting strategies to improve semantic consistency and multimodal expression trust. Second, it incorporates TCM knowledge ontology relation constraints during construction, designs a structural penalty mechanism and role semantic bias terms to ensure the logical rationality of graph nodes and edges and TCM interpretability. Third, it combines graph attention propagation mechanisms and role modeling networks to achieve semantic enhancement and semantic role recognition of the graph structure. Furthermore, it introduces a semantic feedback-driven graph self-evolution strategy at the structural level, automatically completing structure cleaning, edge completion, and edge weight fine-tuning, significantly improving the clinical usability of the graph. Finally, the system supports standardized graph database output and API service deployment, ensuring its calling efficiency and structural update capabilities in real-world diagnostic and treatment systems. Overall, this invention is the first to achieve a complete closed-loop process from raw clinical data, through multimodal modeling, structure generation, graph optimization, role modeling and semantic feedback self-evolution, to deployment and output. It is a highly implementable intelligent graph construction system oriented towards real-world TCM scenarios. Attached Figure Description

[0054] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0055] Figure 1 This is a flowchart of a method for constructing a clinical TCM knowledge graph based on a multimodal graph attention network. Detailed Implementation

[0056] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0057] In one or more embodiments, such as Figure 1 As shown, a method for constructing a clinical TCM knowledge graph based on a multimodal graph attention network is disclosed. The method includes the following steps S1-S6:

[0058] S1. Acquire multimodal data in the clinical environment of traditional Chinese medicine, including: text data, image data and audio data, and map the multimodal data into a multimodal semantic representation set and a modality tag set of the corresponding modality vector; wherein the modality tag set is used to record the modality source of each modality vector.

[0059] Specifically, the goal of this step is to map the multi-modal data from real TCM clinical environment, including text, image and audio, into a shared semantic vector space after customized processing. This not only provides high-quality structured input for the subsequent construction of knowledge graph, but also accurately retains the unique information expression characteristics of TCM, such as symptom combination, tongue diagnosis, oral inquiry, etc.

[0060] Input: Text data T: mainly from TCM electronic medical record system (such as chief complaint, history of present illness, diagnosis results), doctors' description of patients' symptoms, prescription text, etc.

[0061] Image data I: including high-resolution tongue image, facial image, obtained from medical image acquisition equipment;

[0062] Audio data A: original voice dialogue record in TCM inquiry process, usually collected by clinic recording system, needs to be transcribed into text.

[0063] Each entry t in text data T i is input into a Chinese encoder based on BERT fine-tuned on TCM corpus. The model consists of 12 layers of Transformer, each containing 768-dimensional hidden states, which has been pre-trained and fine-tuned on 1 million TCM medical records and prescription annotation corpus. The final output of the model is a context vector embedding f t with dimension d ti , calculated as follows:

[0064]

[0065] where, is the vector representation of the i-th text, from the text input t i , with dimension d t = 768, output by the BERT model fine-tuned on TCM corpus BERT CM .

[0066] Each image i in image data I j is input into a ResNet-50 network with transfer learning, and the final output is a visual feature vector , calculated as follows:

[0067]

[0068] where, i j is the input image, ResNet50 is a 50-layer residual neural network, and GlobalAvgPool represents global average pooling on the feature map, with output vector dimension d i = 512, i.e.

[0069] The voice data A is first transcribed into text by the Wenet model and then fed into the BERT model to obtain the voice semantic representation Its processing mode is equivalent to the text mode.

[0070] Among them, in order to unify the multi-modal vector representation, three groups of modal projection matrices are defined Linear transformation is performed on the three types of modal vectors, and finally all modalities are mapped into the same d-dimensional semantic space, and the mapping formula is as follows:

[0071]

[0072] Among them, m∈{t,i,a} represents the modal type, represents the original vector from the modal m (i.e. ), W m is the projection matrix corresponding to the modal, represents the final unified semantic embedding vector. The linear projection stage realizes the alignment of different modal spaces through training, and in the implementation, the normalization strategy is used to eliminate the dimension of each modal output.

[0073] The innovation of the present scheme lies in not simply concatenating or fusing modal features, but establishing an isomorphic embedding space through structural alignment, so that the subsequent knowledge graph can be connected according to semantic similarity without explicitly distinguishing modalities when constructing.

[0074] The output of this step includes two parts: (1) a set of unified semantic representation vectors F = {f1, f2,..., f n}, where each represents an extraction vector of a clinical entity, which is input as a graph node; (2) a set of modal labels M = {m k}, which marks the original modal source of each f k , which is used for attention weighting processing when constructing the graph structure in the next step.

[0075] S2, obtain the ontology of the traditional Chinese medicine knowledge graph and the standard type, attribute judge all modal vectors in the multi-modal semantic representation set according to the standard type, to obtain the corresponding type label; correspond all modal vectors in the multi-modal semantic representation set to nodes to construct an initial traditional Chinese medicine clinical knowledge graph G0; wherein the edges of the initial traditional Chinese medicine clinical knowledge graph are obtained by connecting the nodes to calculate the connection score.

[0076] Specifically, based on the multi-modal semantic vector output in step one, the knowledge structure ontology in the field of traditional Chinese medicine is combined to construct the initial graph node and its semantic connection relationship, that is, the initial structure graph. The core goal is not only to form a structured data network, but also to realize a graph prototype that is constrained by traditional Chinese medicine knowledge, semantic and modal fusion, and logically self-consistent. The structure will directly determine the propagation path and reasoning accuracy of the subsequent graph neural network, so the correctness, pertinence and explainability of this step are extremely crucial. Especially in the traditional Chinese medicine scene, the knowledge connection between syndrome, prescription, constitution and symptoms has high asymmetry and hierarchy. If the edge is constructed without restriction depending on the similarity of semantic vectors, it will deviate seriously from the actual traditional Chinese medicine diagnosis and treatment logic. Therefore, we propose an innovative three-in-one structure generation mechanism of vector semantic matching + ontology relationship constraint + structure penalty regularization.

[0077] Among them, the input of this step comes entirely from the output results of step one, including two parts:

[0078] Multi-modal semantic representation set F = {f1, f2,..., f n}, where each is the representation vector mapped to the shared semantic space after being encoded by BERT, ResNet or a speech model;

[0079] Modality label set M = {m k}, recording the modality source (text / image / audio) of each f k , used for attention layering in subsequent edge weight calculation.

[0080] First, determine the type of each representation vector f k in F. Instead of using a multi-class classifier for prediction, we perform vector matching with the entity prototype set provided by the ontology of the traditional Chinese medicine knowledge graph. We provide 15 standard types (including: symptoms, syndromes, constitution types, prescriptions, single herbs, treatment methods, etiology and pathogenesis, meridian points, tongue appearance, pulse appearance, chief complaint description, auxiliary examination results, treatment response, health advice and dietary taboos), each type t corresponds to a prototype vector c t , which is obtained by sampling multiple representative word groups from the traditional Chinese medicine standard term library and inputting them into the encoding model (BERT or image / speech to text encoding) in step one. To improve the adaptability to edge entities, we introduce a relative modality awareness term to distinguish the semantic intensity embodied by different modalities. The optimization goal of node type prediction is as follows:

[0081]

[0082] where, f​k is the current node vector, c t is the prototype vector of type t, δ(m k , t) is a modal perception bias term, which is used to express the contribution of modal to entity type confidence, for example, tongue image is more discriminative for tongue symptom; α is the balance parameter of modal perception, the empirical value is 0.05-0.2. The creativity of this design is that it introduces modal information into semantic type discrimination, which embodies the structure design principle of sensory-diagnosis linkage specific to traditional Chinese medicine field in multi-modal scene, different from the conventional argmax matching. is the type label.

[0083] where, after completing the type attribution judgment, we will all f k corresponding to the node v k , constitute the node set V0={v k}, each node is attached to the triple Then enter the edge relationship construction link.

[0084] Further, in order to construct the edge set E0, we introduce the entity connection rules given in traditional Chinese medicine ontology on the basis of considering semantic similarity This set contains the following standard connection types: syndrome type-presentation-symptom, syndrome type-applicable-constitution type, syndrome type-use-treatment method, syndrome type-recommend-recipe, recipe-contains-single herb, single herb-acts on-meridian site, symptom-presentation-tongue image, symptom-presentation-pulse image, chief complaint description-associated-symptom, auxiliary examination result-suggests-etiology and pathogenesis, treatment method-causes-treatment reaction, treatment method-reference-dietary taboos, health advice-associated-constitution type, tongue image-assist in judging-syndrome type, pulse image-assist in judging-syndrome type. But only these two can not effectively control the semantic drift phenomenon (such as yin deficiency and fire and qi deficiency may be similar in sentence meaning but are actually different syndrome types). Therefore, we design an edge connection objective function, which adds a structure penalty term on the basis of preserving the similarity term Optimize the connection score function as follows:

[0085]

[0086] where, cos(f i , f j ) is the cosine similarity of f i and f j , indicates whether it belongs to the allowed type pair, is the structure penalty function, if f i and f jThe type combination history of the belonging type combination has a low connection density (i.e., is rare in an existing traditional Chinese medicine graph), and a greater penalty is given; λ is a hyperparameter (recommended value 0.1-0.5) for controlling the penalty intensity. For example, Sijunzi Decoction and diarrhea may be semantically close in the training corpus, but lack a direct connection logical basis in the traditional Chinese medicine knowledge system, and are thus filtered by the penalty term.

[0087] Further, to improve the interpretability of the edge connection confidence, a modality weighting mechanism is introduced for the semantic similarity between different modalities. When constructing the final edge e ij , if two nodes are from different modalities, such as a chief complaint described in speech and a tongue image, the system will adjust the weight according to the empirical reliability of the modality combination. For example, a tongue image + tongue description will get a higher edge weight, and a medical history text + tongue image will be suppressed.

[0088] The overall creativity of this step is reflected in three aspects: 1) nonlinearly fusing the ontology structure and semantic similarity to avoid overfitting to surface semantics in the construction phase; 2) introducing a modality-aware matching bias to establish a reasonable trust relationship between the graph structure and the data source, and solving the problem of multi-modal semantic misalignment; 3) dynamically avoiding false connections caused by semantic drift through a structure penalty mechanism, and ensuring the quality of the graph from a structural level.

[0089] The output of this step includes two structures: one is a node set V0={v k}, where each node contains a semantic vector f k , a type label and a modality label m k ; the other is an edge set E0={e ij}, which is composed of all connections with a score S ij greater than a threshold τ, and each edge is accompanied by a direction, a relationship type label and a construction score. This structure will serve as the input basis for the graph neural network optimization module.

[0090] S3, performing attention enhancement based on a graph neural mechanism on the initial traditional Chinese medicine clinical knowledge graph G0 to generate a structure-enhanced graph G1.

[0091] Specifically, the core goal of this step is to further optimize the semantic expression and structural correlation of nodes based on the initial TCM clinical knowledge graph structure G0=(V0, E0) constructed in the previous stage through graph neural mechanisms, thereby forming a graph G1 with stronger context semantic recognition ability and cross-modal expression ability. In the TCM scenario, the real clinical relationships between knowledge entities are far more than just the connectability on the structure level, and also include complex features such as subtle semantic shifts between syndromes, differences in the same symptom under different modalities, and multiple weight logic of edge relationships. Traditional graph attention mechanisms cannot effectively capture the coupling structure between these heterogeneous medical semantics. Therefore, we innovatively propose a multi-modal semantic-structure collaborative attention mechanism, which combines modal perception, edge confidence regulation, and semantic distribution sparsity regulation strategies, so that the graph neural network updates the node representation while adhering to the structure topology and embodying the TCM semantic logic.

[0092] Among them, the output of the complete step two is input into this step, including:

[0093] Node set V0={v k}, each node v k is a triple Wherein is the embedding vector in the unified semantic space (unified mapping after encoding text, image or voice respectively), is the type label matched from the TCM knowledge ontology, m k represents the modal source (text / image / audio);

[0094] Edge set E0={e ik}, each edge e ik is a structural connection between nodes v i and v j , carrying edge type (such as composition, manifestation, etc.) and construction confidence S ij (originating from the structure scoring of step two);

[0095] Optional ontology auxiliary matrix records the semantic strength and connection stability of each edge type, which is used for regularization control (this matrix is generated by historical graph statistics, not model learning).

[0096] Further, on the basis of the traditional graph attention network (GAT), we have made three modifications to its attention score calculation method: (1) Introducing a modal attention adjustment term, so that the asymmetric trust relationship between modalities is explicitly modeled; (2) Introducing a structure confidence weight term, so that high-confidence connections have a greater contribution to the representation; (3) Adding a semantic distribution regulation regularization term to alleviate the bias of high-frequency symptom or drug nodes in edge generation.

[0097] First, we calculate the attention score α ij for each adjacent node pair (v i ,v j ) as follows:

[0098]

[0099] where, is the linear transformation matrix of node features, which uniformly adjusts the representation space of the original semantic vector; is the attention score vector; ψ(m i ,m j ) is a modal compatibility function that represents the credibility of modal combination with empirical weights (e.g., image-image is set to 1, text-image is set to 0.6, etc.); S ij is the structural confidence score output in step two; R ij is a semantic frequency penalty factor defined as a function of the sum of the frequencies of the concepts corresponding to the two nodes in the graph, specifically R ij = log(1+freq(v i )+freq(v j )); β, γ, η are three control item hyperparameters that control modal bias, structural confidence, and frequency suppression strength (recommended initial values are 0.2, 0.3, and 0.1); || represents vector concatenation.

[0100] It is important to note that the three innovations of this formula are as follows:

[0101] Adding ψ(m i ,m j ) quantifies the modal credibility as attention weight, which is crucial when dealing with traditional Chinese medicine data, such as tongue image description and tongue image, which have different levels of authority;

[0102] Introducing S ij as a structural confidence adjustment term to prevent edge weights from diluting the strong structural information in the ontology during graph neural network propagation;

[0103] The regularization term R ij suppresses the bias of high-frequency entity dominance in propagation, especially for high-frequency terms such as spleen deficiency and Huangqi in traditional Chinese medicine, to avoid them becoming the semantic center of the attention network.

[0104] Further, after completing the attention score calculation, the node representation update method uses the standard attention weighted aggregation, i.e.:

[0105]

[0106] where f′ i represents the node v iσ is the activation function, ELU or GELU is recommended to capture the nonlinear structure boundary; W is consistent with the above formula, representing the node v i adjacent set.

[0107] The entire network can be stacked with multiple layers to achieve multi-order propagation (two layers are recommended, and each layer has independent training parameters). It should be pointed out that in the training process, the initial label of the node can be used as an auxiliary loss in supervised training, but the design principle of this step is to enhance the semantic consistency in the structure propagation, rather than direct classification.

[0108] The output of this step is the graph G1=(V1,E1,A1) after structure enhancement, where V1 contains the updated representation f′ i of each node; E1 uses the structure edge of the previous stage; A1 is the attention weight matrix α ij of each edge in the final graph attention propagation process. This output will be directly used in the next stage of semantic role identification and structure normalization processing, which is the key foundation to realize the explainable reasoning ability of the graph.

[0109] S4, obtaining a standard role set of a traditional Chinese medicine knowledge base, performing semantic role identification on each node in the structure-enhanced graph G1 based on the standard role set, to generate a node set with role labels, and updating the structure-enhanced graph G1 to obtain a normalized graph G2.

[0110] Specifically, the main goal of this step is to identify and model the semantic role of each node in its context based on the graph structure G1 output by step three, such as the common roles of main and secondary symptoms, monarch drugs, and ministerial drugs in traditional Chinese medicine clinical practice. At the same time, the graph structure is reasonably normalized according to the role distribution.

[0111] Further, unlike traditional graphs that only distinguish nodes by type labels, a symptom in traditional Chinese medicine semantics may play different roles in different contexts. If this context-sensitive role is not modeled, it will lead to directional errors or weight biases in graph structure learning. Therefore, this step introduces a lightweight role modeling network M r that identifies the semantic role of each node in a specific context graph during the propagation of node features, and uses role label information to clean up and normalize possible semantic conflicts in the structure, thereby forming a semantic-enhanced graph with clinical interpretability.

[0112] This step is based on the output of step three, and the input includes:

[0113] Graph structure G1=(V1,E1,A1), where V1={v k} is the node set, and each node vk contains updated semantic representation vector f' k (optimized by multi-modal attention mechanism), type label (modality label m (such as symptoms, prescriptions, syndromes, etc.), modality label m k (such as images, text, voice);

[0114] Edge set E1 is a preliminary structure edge between nodes, including connection direction and relationship label;

[0115] Edge weight matrix A1 = {a ij} is the weight of each edge in the graph attention propagation process;

[0116] Auxiliary vocabulary from traditional Chinese medicine knowledge base, including standard role set, such as {main symptoms, auxiliary symptoms, monarch drugs, minister drugs, assistant drugs, and messenger drugs}.

[0117] The role recognition task is to identify the semantic function of each node in the current graph based on its representation f' k . We use a reproducible context-aware role modeling module M r , which is a two-layer neural network, with f' k as input, the first layer being a 128-dimensional fully connected layer with ReLU activation, and the second layer being a Softmax layer outputting role distribution probability. The training data comes from structured annotated traditional Chinese medicine graphs (such as “Traditional Chinese Medicine Diagnosis and Treatment Knowledge Ontology” and real clinical prescription data sets), and the supervised labels are constructed by the roles played by nodes in real graphs.

[0118] Further, in order to make the role recognition have context-aware ability, we also model the adjacent information of each node as part of the input, with the specific input being the concatenated features where represents the average value of the neighbor node representation of the node. The role prediction loss function is cross-entropy plus edge conflict regularization:

[0119]

[0120] where CE(·) is the cross-entropy loss, y k is the target role label of node v k ; r i , r j is the predicted role label of node v i , v j ; is an indicator function, which is 1 if the roles of two nodes should not be logically connected (such as main symptoms-messenger drugs); a ijis the attention weight of the current edge; λ is the penalty coefficient. This mechanism ensures that the role labels learned by the model have logical interpretability at the structural connection level, avoiding semantic error path reinforcement.

[0121] For example: Huangqi in Sijunzi Decoction is usually the monarch, Baizhu is the minister, and Gancao is the assistant. If the system cannot distinguish the functional hierarchy among these drugs in semantic features, but through their adjacency in the graph structure (such as the difference in connected symptom nodes, syndrome nodes), it can be identified that Huangqi aggregates more high-weight main symptom nodes, so it is identified as the monarch by the model.

[0122] After the role label output is completed, we perform normalization processing on the structure edge E1: for edge (v i ,v j ), if its two end node role label combination is identified as semantic mismatch (such as main symptoms pointing to auxiliary drugs), and its weight α ij is less than the set threshold τ, it is excluded from the edge set. Conversely, if the node combination is of semantic complementary type (such as main symptoms-monarch, monarch-syndrome), the weight of the edge is increased by no more than 0.15, to ensure that it is given priority in subsequent reasoning.

[0123] The core innovation of this step is reflected in three aspects: (1) Introducing role modeling mechanism to identify the functional semantic status of nodes in the graph, which is missing in existing multi-modal graph systems; (2) Role modeling is based on context, dynamically judging semantic function based on adjacent structure, rather than fixed dictionary hard matching, improving adaptability; (3) Combining role conflict logic to soft-prune edge structure, preserving the original structure while achieving semantic normalization.

[0124] The output of this step includes: (1) Node set V2 = {(v k ,r k )} with role labels, where r k is the predicted role of node v k ; (2) Structure normalized graph G2 = (V2, E2, A2), where E2 is the edge set after role logic cleaning, and A2 is the updated edge attention weight matrix.

[0125] S5, calculating the edge importance score based on the normalized graph G2, and selecting the final edge structure based on the edge importance score to generate the standardized graph structure G final .

[0126] Specifically, this step aims to model the self-evolution of the graph structure based on the semantic role annotation and structure normalization results of step four, driven by semantic consistency.

[0127] This step uses the output of Step Four as input, specifically:

[0128] The normalized graph G2 = (V2, E2, A2), where V2 = {v k} and each node v k contains a semantic vector Type label Role label r k (such as main symptoms, monarch drugs, etc.);

[0129] Edge set E2 = {e ij} and each edge records the connection type and direction;

[0130] Edge weight set A2 = {a ij} calculated by the graph attention network in Step Three, which has been updated according to the role logic part in Step Four;

[0131] Ontological role relationship dependency matrix From the standard Chinese medicine knowledge base, defining the semantic connection strength between roles, for example, the role pair strength of main symptoms-monarch drugs is set to 1, and the role pair strength of main symptoms-adjunct drugs is set to 0.2.

[0132] Further, the graph structure self-evolution modeling task is defined as a fine-tuning process for the edge set E2, i.e., determining whether each edge should be retained, strengthened, or weakened, and appropriately filling in potential structural edges missed due to role combination. Unlike general graph structure learning, we specifically introduce an edge importance function based on semantic difference and role compatibility as the optimization basis.

[0133] First, for each edge e ij , its role consistency contribution and semantic structure sparsity contribution are calculated, and then the edge importance score S ej is generated together:

[0134]

[0135] where f′ i ,f′ j are the latest semantic vectors of the nodes; cos(f′ i ,f′ j ) is the semantic similarity; represents the structural trust of the role combination r i ,r j ; d i ,d j are the graph degrees (connection edge numbers) of the current nodes; is the average degree of the nodes in the graph; λ is a weight parameter (recommended value 0.1-0.2) used to balance the influence of structural sparsity on optimization.

[0136] Further, the core innovation of the formula is to model the connection behavior of each edge in the graph as a semantic-structural joint contribution, considering not only content similarity but also measuring the credibility on the structure according to the knowledge of TCM role combination, while penalizing nodes with excessive connections to improve the sparsity and clarity of the graph.

[0137] According to S ij , we design the following update strategy:

[0138] If S ij < θ1 (e.g. 0.3), the edge is marked as low contribution, and the system linearly decays its edge weight from α ij to 0.1·α ij .

[0139] If S ij > θ2 (e.g. 0.7) and the edge is not in the structure yet, it is automatically added to E3 with edge weight α ij = 0.3.

[0140] Otherwise, the original edge and edge weight are kept.

[0141] Example: The node v i of spleen deficiency syndrome has a high semantic similarity with the node v j of fatigue, and r i = "syndrome type", r j = "main symptom", and S ij will be high. Even if the graph is not originally connected, it can be supplemented under the condition.

[0142] For high-frequency symptom or drug nodes, we also design a structural inhibition mechanism. If d i exceeds , i.e. the connection is too dense, no matter how high S ij is, it will not be enhanced to avoid overfitting common nodes.

[0143] The final edge structure E3 is the structure after deletion, enhancement, and completion under the above rules, and A3 is the final weight matrix of all retained edges or newly created edges. It should be noted that this step does not involve new supervised learning or external tasks, but only uses the existing semantic and role information from the previous stage to build a structural evolution model, completely closed-loop within the modeling level, and in line with the patent structure task boundary.

[0144] The output includes: the graph G final = (V2, E3, A3) after structural self-evolution, where the node set V2 remains unchanged, the edge set E3 is the edge set after processing by the structural evolution mechanism, and A3 is the corresponding edge weight. This graph will enter the standardization and deployment process in the next step as a deliverable TCM clinical knowledge graph product.​

[0145] S6. The standardized map structure G final Import the graph database and build a RESTful interface to access clinical knowledge.

[0146] Specifically, this step, as the final stage of the entire patent process, aims to deploy the knowledge graph as a service entity that can be directly accessed and invoked by traditional Chinese medicine diagnosis and treatment systems after completing the optimization of the knowledge graph structure and semantic standard modeling.

[0147] The input for this step is the output structure of step five:

[0148] Standardized spectral structure G final = (V2, E3, A3), each node v k Includes semantic representation f′ k Entity type Character tag r k Modal number m k ;

[0149] The structure is unified and marked with fields, including semantic consistency flags and modality standard mapping fields (which have been completed and validated in step 5);

[0150] Atlas Application Template Define the functions to be invoked in the deployment scenario, such as intelligent syndrome matching, prescription interpretation, and symptom diagnosis tracking.

[0151] First, the structured map data G final Import the graph database. To ensure operational efficiency and structural availability, we chose a graph database system that supports attribute graphs and edge weights (such as Neo4jEnterprise or TigerGraph). The graph node information V2 is transformed into a set of execution nodes, each containing standard fields, such as:

[0152] id: Unique identifier for the node; label: Standard terminology name for the node; type: Type label (e.g., symptom, syndrome, prescription); role: Semantic role (e.g., principal symptom, principal drug); modality: Modality number; vector: Semantic vector (optionally enabled).

[0153] Edge set E3 is converted into a relation type of the graph database, while retaining edge weights A3. If there are semantic inconsistency markers in an edge, they are set to false as edge attribute "valid", used only for backend analysis and not involved in the call process.

[0154] Secondly, the system follows The defined TCM application function constructs a service interface. Each typical clinical task (such as inputting a syndrome type, returning possible symptoms and weights, inputting a prescription, and analyzing its core and auxiliary drug combinations) is encapsulated as a RESTful API interface, and the interface internal logic depends on the path query and edge weight accumulation algorithm of the graph database.

[0155] For example: input spleen yang deficiency as a syndrome entity, the system will perform the following operations:

[0156] Query node v k , which satisfies = syndrome type and label = spleen yang deficiency;

[0157] Traverse all outgoing edges {e kj}, and filter the target nodes with edge type as performance and a kj >0.3;

[0158] Return the label of node v j (such as fatigue and loose stool) and the corresponding edge weight as the matching confidence.

[0159] This mechanism is the embodiment of graph atlas practical application, and through interface log real-time record calling behavior, providing basis for subsequent maintenance and model iteration.

[0160] Finally, to ensure the integrity of system deployment, the graph service supports:

[0161] Modular update (allowing only the update of prescription-drug subgraph);

[0162] Modality split view call (such as returning only the structure subgraph generated by the image modality);

[0163] Multi-model docking (supporting docking with semantic retrieval model and dialogue decision system for use);

[0164] The output of this step is a serviceable knowledge graph system with callable ability, which includes:

[0165] A deployed and running graph database instance (containing node, edge, and weight information in G final );

[0166] A set of external RESTful interfaces that can support all knowledge call requests in the diagnosis and treatment system;

[0167] Log recording and structure feedback module for later graph update and effect evaluation.

[0168] The entire patent system completes the full closed loop from data collection-structure generation-semantic optimization-semantic modeling-structure evolution-practical deployment, achieving the goal of clinical TCM knowledge graph construction method based on multi-modal graph attention network.

[0169] The above is the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the principles of the present application, can also make several improvements and refinements, these improvements and refinements are also considered the scope of protection of the present application.

Claims

1. A method for constructing a clinical traditional Chinese medicine knowledge graph based on a multi-modal graph attention network, characterized in that, The method comprises: Obtaining multi-modal data in a traditional Chinese medicine clinical environment, including text data, image data and audio data, and mapping the multi-modal data into a multi-modal semantic representation set and a modal label set of corresponding modal vectors respectively; wherein the modal label set is used to record the modal source of each modal vector; Obtaining a traditional Chinese medicine knowledge graph ontology and standard types, performing attribution judgment on all modal vectors in the multi-modal semantic representation set according to the standard types to obtain corresponding type labels; and corresponding all modal vectors in the multi-modal semantic representation set to nodes to construct an initial traditional Chinese medicine clinical knowledge graph G0; wherein the edges of the initial traditional Chinese medicine clinical knowledge graph are obtained through connection score calculation on the nodes; Performing attention enhancement based on a graph neural mechanism on the initial traditional Chinese medicine clinical knowledge graph G0 to generate a structure-enhanced graph G1; Obtaining a standard role set of a traditional Chinese medicine knowledge base, performing semantic role recognition on each node in the structure-enhanced graph G1 based on the standard role set to generate a node set with role labels, and updating the structure-enhanced graph G1 to obtain a normalized graph G2; According to the normalized graph G2, an edge importance score is calculated, and based on the edge importance score, a final edge structure is screened to generate a standardized graph structure G final ; The standardized graph structure G final Imported into a graph database, RESTful interfaces are constructed to enable clinical knowledge invocation.

2. The multi-modal graph attention network-based clinical traditional Chinese medicine knowledge graph construction method according to claim 1, characterized in that, The step of mapping the multi-modal data into a multi-modal semantic representation set and a modal label set of corresponding modal vectors respectively comprises: Each entry in the text data is input into a Chinese encoder of a BERT model fine-tuned based on traditional Chinese medicine corpus to generate a vector representation of the corresponding text; Each image of the image data is input into a ResNet-50 network subjected to transfer learning, and finally output as a visual feature vector of the corresponding image; The speech data is first transcribed into text by a Wenet model, and then input into the BERT model fine-tuned based on traditional Chinese medicine corpus to obtain a speech semantic representation of the corresponding text; Three groups of modal projection matrices are defined for the text data, image data and audio data respectively, and linear transformation is performed on the three groups of modal projection matrices respectively, finally all modalities are mapped to the same d-dimensional semantic space to obtain the modal label set, represented as: where m e {t, i, a} denotes the modality type, denotes the original vector from modality m, W m is the projection matrix for the corresponding modality, denotes the final unified semantic embedding vector; wherein the set of modality labels is represented as M = {m k} labels each f k original modality source.

3. The multi-modal graph attention network-based clinical traditional Chinese medicine knowledge graph construction method according to claim 1, characterized in that, The standard types include symptoms, syndromes, constitution types, prescriptions, single herbs, treatment methods, etiology and pathogenesis, meridian positions, tongue appearances, pulse appearances, chief complaint descriptions, auxiliary examination results, treatment responses, health advice and dietary taboos.

4. The multi-modal graph attention network-based clinical traditional Chinese medicine knowledge graph construction method according to claim 1, characterized in that, The step of obtaining a traditional Chinese medicine knowledge graph ontology and standard types, and performing attribution judgment on all modal vectors in the multi-modal semantic representation set according to the standard types to obtain corresponding type labels comprises: Performing type determination on each modal vector in the modal label set based on the standard types to obtain a predicted type label; wherein each type corresponds to a prototype vector; All modal vectors f k Corresponding to the node v k , constitute a node set V0={v k}, each node is attached to a triple For the prediction type label; Introducing entity connection rules in traditional Chinese medicine ontology to all modal vectors f k Perform cosine similarity calculation to obtain the initial connection score of modal vector f k , filter the modal vectors with initial connection scores greater than a preset threshold to constitute a corresponding edge set E0={e ij}, e ij is the edge of corresponding modal i and j; Based on the node set V0={v k} and the edge set E0={e ij}, an initial traditional Chinese medicine clinical knowledge graph structure G0=(V0, E0) is obtained.

5. The multi-modal graph attention network-based clinical traditional Chinese medicine knowledge graph construction method according to claim 1, characterized in that, The step of performing attention enhancement based on a graph neural mechanism on the initial traditional Chinese medicine clinical knowledge graph G0 to generate a structure-enhanced graph G1 comprises: For the initial TCM clinical knowledge graph G0, for each pair of adjacent nodes (v i ,v j Calculate its attention score α ij , represented as: wherein, is a node-specific linear transformation matrix, which uniformly adjusts the representation space of the original semantic vector; is an attention score vector; ψ(m i , m j ) is a modality compatibility function, which represents the credibility of the modal combination with empirical weights; S ij is an initial connection score; R ij is a semantic frequency penalty factor, which is defined as a function of the sum of the frequencies of the concepts corresponding to the two nodes in the graph, specifically R ij = log(1 + freq(v i ) + freq(v j )); β, γ, η are three control item hyperparameters, which respectively control the modal bias, structural confidence, and frequency suppression intensity; || represents vector splicing; After completing attention score calculation, the node representation update method adopts standard attention weighted aggregation to obtain the updated representation of the node; Based on the node update representation, multi-order propagation is performed to obtain a structure-enhanced graph G1.

6. The multi-modal graph attention network-based clinical traditional Chinese medicine knowledge graph construction method according to claim 1, characterized in that, The standard role set of the traditional Chinese medicine knowledge base is obtained, semantic role recognition is performed on each node in the structure-enhanced graph G1 based on the standard role set, a node set with role labels is generated, and the structure-enhanced graph G1 is updated to obtain a normalized graph G2. The steps specifically include: inputting the representation of each node in the structure-enhanced graph G1 into a pre-constructed double-layer neural network, outputting a predicted role of the corresponding node; the loss function of the double-layer neural network is represented as: where CE(·) is the cross-entropy loss, y k is the target role label of node v k ; r i is the predicted role label of node v j ; v i , j ; is an indicator function, which is 1 if the roles of two nodes should not be connected logically; a ij is the attention weight of the current edge; l is a penalty coefficient; f′ k is the representation of the node in the structure-enhanced graph G1; AGG(N k ) is the average of the representations of the neighboring nodes of the current node. Based on the predicted role, the structure edge E1 of the structure-enhanced graph G1 is normalized to generate a normalized edge; In combination with the predicted role and the nodes of the structure-enhanced graph G1, a node set with role labels is generated, and in combination with the normalized edge, a normalized graph G2 is obtained.

7. The multi-modal graph attention network-based clinical traditional Chinese medicine knowledge graph construction method according to claim 6, characterized in that, The filtering step of normalizing the structure edge E1 of the structure-enhanced graph G1 based on the predicted role to generate a normalized edge includes: For the edge (v) of the structure-enhanced graph G1 i ,v j If the combination of the role labels of its two end nodes is It is determined to be a semantic mismatch, and its current edge's attention weight α ij If the value is less than a set threshold, it will be removed from the set of structural edges.

8. The multi-modal graph attention network-based clinical traditional Chinese medicine knowledge graph construction method according to claim 1, characterized in that, The step of calculating edge importance scores according to the normalized graph G2, and screening out final edge structures based on the edge importance scores to generate a standardized graph structure G final specifically comprises: The cosine similarity of the nodes of the normalized graph G2 is measured and adjusted based on the predicted role, and the current node degree of the normalized graph G2 is normalized and averaged to obtain an edge importance score; According to the edge importance score, the normalized graph G2 is updated, and the update strategy is as follows: If the edge importance score is less than a first preset threshold, the edge is marked as low contribution, and the system sets the attention weight of the current edge to 0 ij Linearly decays to 0.1 · α ij ; If the edge importance score is greater than a first preset threshold, and the edge has not been in the structure, a new edge is automatically supplemented and the attention weight of the new edge is set to 0.3; If there are other conditions, the original edge and the edge weight are reserved; generating a standardized graph structure G based on the update policy final .

9. The multi-modal graph attention network-based clinical traditional Chinese medicine knowledge graph construction method according to claim 1, characterized in that, The standardized graph structure G final The step of importing into a graph database and constructing a RESTful interface to enable clinical knowledge invocation specifically includes: transforming nodes of the standardized graph structure G final into a set of execution nodes, each execution node containing standard fields; and transforming edges of the standardized graph structure G final into relation types of the graph database, and preserving corresponding edge weights; According to the traditional Chinese medicine application function of the graph application template, a service interface is constructed, wherein each typical clinical task is encapsulated as a RESTful API interface, and the internal logic of the interface depends on the path query and edge weight accumulation algorithm of the graph database; wherein the graph application template is used to define the calling function in the deployment scene; the RESTful API interface supports view calling and modular updating according to modalities.

10. The multi-modal graph attention network-based clinical traditional Chinese medicine knowledge graph construction method according to claim 6, characterized in that, The structure of the double-layer neural network is as follows: The first layer is a 128-dimensional fully connected layer using ReLU activation, and the second layer is a Softmax layer.

Images