A medical language model evaluation method and device based on dynamic cost map matching and reasoning path logic review

Abstract

This invention discloses a method and apparatus for evaluating medical language models based on dynamic cost graph matching and inference path logic review, belonging to the interdisciplinary field of artificial intelligence and medical informatics. The proposed technical solution includes: constructing a knowledge graph; receiving the output content of the medical language model under test and parsing it into a test graph; vectorizing and optimizing the node text information of the knowledge graph and the test graph; calculating the graph edit distance between the test graph and the knowledge graph based on a graph edit distance algorithm, using dynamically calculated node replacement costs, converting it into a similarity score, normalizing it, and then calculating the local and global similarity scores between the test graph and the knowledge graph; using a depth-first search algorithm to detect logical loops, or detecting semantic coherence by calculating the embedding vector similarity of adjacent nodes on the path; and finally generating an evaluation report. This invention achieves automated, multi-dimensional, and in-depth evaluation of the output content of medical language models.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and medical informatics, and specifically relates to a method for high-precision calculation of the similarity of medical knowledge graphs, and more particularly to a method and apparatus for evaluating medical language models based on dynamic cost graph matching and inference path logic review. Background Technology

[0002] In recent years, large language models have made significant progress in the field of natural language processing and have demonstrated powerful capabilities in tasks such as open-domain dialogue and text generation. However, how to scientifically, efficiently and reliably evaluate the output quality of these models has always been a core challenge in this field. This challenge is particularly prominent and urgent in high-risk fields such as medicine, law, and finance.

[0003] Currently, there are two main types of evaluation methods for large language models, but both are seriously inadequate in their applicability in the medical field.

[0004] 1. Expert-based evaluation methods. This method typically involves medical experts subjectively evaluating the completeness, accuracy, safety, and logical consistency of the model's output based on their professional knowledge, clinical guidelines, and experience. While this is currently the accepted standard, its drawbacks are significant: First, it is extremely costly, as the time commitment of senior medical experts limits the scale and frequency of evaluations; second, it is inefficient, struggling to provide rapid feedback on the massive amounts of output generated during model iterations; and finally, evaluation standards are difficult to standardize, as subjective differences among experts lead to poor consistency and reproducibility of evaluation results, making it difficult to support large-scale, objective model comparisons and optimizations.

[0005] 2. Automated Evaluation Methods. To overcome the limitations of manual evaluation, the industry widely adopts automated metrics. Among them, metrics such as BLEU and ROUGE, based on n-gram matching, are designed to measure the surface affix overlap between machine translations or text summaries and reference translations. However, directly applying these metrics to medical text evaluation suffers from a fundamental flaw: measurement distortion. They can only capture the fluency and lexical similarity of the text, completely failing to determine the correctness of medical facts, the rigor of clinical reasoning, and the safety of treatment plans.

[0006] The limitation of existing technologies lies in their inability to address the core dimensions of medical text evaluation:

[0007] 1. Lack of semantic accuracy: The model may generate a text with professional vocabulary and fluent sentences, but the description of the etiology, drug mechanism or surgical indications may contradict authoritative medical knowledge. Existing automated indicators lack the ability to identify such fallacies.

[0008] 2. Collapse of the internal logical structure: Medical reasoning follows strict logical chains, and existing evaluation methods are almost powerless to address deep structural problems such as circular arguments and logical jumps in the text. For example, a model might assert that "the patient has diabetes because their blood sugar level is high," but then claim elsewhere that "the reason the patient's blood sugar is high is diabetes," constituting a circular argument, which is fatal in clinical decision-making.

[0009] In summary, the field has long lacked an automated evaluation tool capable of simultaneously penetrating the three dimensions of semantics, structure, and logic. This deficiency not only slows down the development of high-quality medical models but also significantly hinders their safe and reliable application in real-world clinical decision support environments. Therefore, developing an evaluation method capable of comprehensively considering model outputs in a multi-dimensional, in-depth, and automated manner has become an urgent technical bottleneck and a widespread industry need for promoting the in-depth development of AI in the medical field. Summary of the Invention

[0010] To address the problems mentioned in the background art, this invention provides a medical language model evaluation method based on dynamic cost graph matching and inference path logic review. This method is based on a pre-defined authoritative medical reference knowledge graph, and its technical solution includes the following steps:

[0011] S1: Construct a knowledge graph containing reference entity nodes and reference relationship edges;

[0012] S2: Receive the output content of the medical language model under test, parse the output content into a graph under test containing entity nodes and relation edges, vectorize the text information of the nodes of the knowledge graph and the graph under test, and use graph neural network to fuse the topological structure information of the graph for vector optimization.

[0013] S3: Calculate the replacement cost of replacing nodes in the test graph with nodes in the knowledge graph using the graph editing distance algorithm, convert the replacement cost into a similarity score, and normalize it. Then, calculate the comprehensive similarity score between the test graph and the knowledge graph through dynamic scoring and multi-round interaction mechanism.

[0014] S4: Perform a logical review on the reasoning path in the graph to be tested. The logical review includes at least one of the following: using a depth-first search algorithm to detect logical loops, or detecting semantic coherence by calculating the similarity of the embedding vectors of adjacent nodes on the path.

[0015] S5: Generate an evaluation report based on the similarity calculated in step S3 and the logical review results in step S4.

[0016] In step S1, a knowledge graph containing reference entity nodes and reference relationship edges is constructed.

[0017] Specifically, simulated doctor-patient dialogue data is generated through a large language model, and based on this, medical NLP tools are used to automatically extract structured triples of "entity-relationship-entity". In order to eliminate the ambiguity of medical terminology and ensure the interoperability of knowledge, all extracted entities are mapped to the international standard medical terminology system.

[0018] The structured triples are constructed as a directed attribute graph, where nodes represent standardized medical entities and edges represent predefined medical relationships; both nodes and edges are attached with structured attributes, including information such as encoding, type, confidence level, and knowledge traceability.

[0019] In the knowledge graph, a predefined standard evidence-based path is a sequence of nodes and edges representing a standard diagnosis and treatment process, which is based on authoritative clinical practice guidelines or mined from a large number of medical documents. An additional key-value pair is added to the edge attributes of these paths to characterize the typical diagnosis and treatment process in the authoritative clinical guidelines.

[0020] In step S2, the output content of the medical language model to be tested is received, and the output content is parsed into a graph to be tested containing entity nodes and relation edges. The text information of the nodes of the knowledge graph and the graph to be tested is vectorized, and the graph neural network is used to fuse the topological structure information of the graph for vector optimization.

[0021] The medical language model under test is constrained to output a triple array in JSON format, or when the output is unstructured natural language, it is extracted and standardized using a medical natural language processing toolkit.

[0022] An array of JSON triples, where each element is an object representing a knowledge triple. This object contains three properties: source (source entity), relation (relation), and target (target entity).

[0023] When the medical big language model under test outputs unstructured natural language text, entities and relationships are extracted from the text through an instant NLP processing pipeline similar to that used when building a knowledge graph.

[0024] The extracted entities undergo a standardization process that is completely consistent with the knowledge graph, namely, querying the standard coding system to obtain their standardized representation, using the standardized entities as nodes and the extracted relationships as edges to construct the graph to be tested.

[0025] By using a pre-trained model, for any node in the graph, its text attributes are concatenated into a text, and this text is input into the pre-trained model to obtain a node embedding vector of fixed dimensions.

[0026] The node attributes are concatenated with the node embedding vector to form the initial complete feature vector before entering the graph attention network;

[0027] By using graph attention networks to fuse graph topological information, the initial complete feature vector is optimized to obtain a final embedding vector that includes the textual information of the nodes and their connection relationships in the graph.

[0028] Optimizing the initial complete feature vector specifically includes: performing a linear transformation on the input features of each node through a shared weight matrix to enhance the node features, and calculating the attention coefficient between nodes;

[0029] The attention coefficients of all nodes are normalized using the softmax function. By using the normalized attention weights, the transformed features of all the neighboring nodes of a node are weighted and summed. Finally, a non-linear activation function is applied to obtain the new feature vector of the node.

[0030] By using a multi-head attention mechanism to capture diverse information, the feature vectors output by multiple heads are concatenated in the intermediate layer, and the average value of the elements of the output vectors of multiple heads is taken in the output layer. This maximizes the preservation and transmission of information diversity within the model, ultimately producing a highly integrated and robust node representation.

[0031] By stacking multiple layers of GAT networks, an L-layer model aggregates neighborhood features through L times, enabling the final representation of each node to capture and integrate the structural and semantic features of all higher-order neighborhoods within its L hops, thus achieving deep integration of information from local to global.

[0032] In step S3, based on the graph edit distance algorithm, the graph edit distance between the graph to be tested and the knowledge graph is calculated by dynamically calculating the node replacement cost. The graph edit distance is converted into a similarity score and normalized. Then, the local similarity score and global similarity score between the graph to be tested and the knowledge graph are calculated through dynamic scoring and multi-round interaction mechanism.

[0033] Based on the graph editing distance algorithm, the node replacement cost is calculated by weighting the first component and the second component. The first component is determined based on the semantic similarity between the node to be tested and the reference node, and the second component is determined based on whether the node type of the node to be tested is consistent with the node type of the reference node.

[0034] Other editing operations have costs, including the cost of adding or deleting nodes and the cost of adding or deleting edges, which are all constants.

[0035] By comprehensively considering the costs of node replacement, node addition / deletion, and edge addition / deletion, the minimum total cost required to convert the graph under test into a knowledge graph is calculated. The minimum total cost is the graph edit distance value.

[0036] The calculated graph edit distance value is converted into a similarity score between 0 and 1, and then normalized.

[0037] After each user input, the local similarity score for that round is calculated in real time, and the global similarity score is updated using a weighted average method.

[0038] In step S4, the reasoning path in the graph to be tested is logically reviewed. The logical review includes at least one of the following: using a depth-first search algorithm to detect logical loops, or detecting semantic coherence by calculating the similarity of the embedding vectors of adjacent nodes on the path.

[0039] The method of detecting logical loops using a depth-first search algorithm includes: systematically traversing every node and edge in the reasoning path graph generated by the medical language model under test to identify whether circular arguments exist; during the depth-first search traversal, a recursive stack recording all nodes on the current traversal path is maintained to detect loops; if when visiting an adjacent node, it is found that the adjacent node already exists in the current recursive stack, then the reasoning path is determined to contain a logical loop.

[0040] Semantic coherence detection by calculating the similarity of the embedding vectors of adjacent nodes on the path includes: calculating the similarity of the embedding vectors of adjacent nodes on the reasoning path; if the similarity is lower than the similarity threshold, it is determined to be a mid-path correction; searching for a reasonable path from the starting point to the ending point in the knowledge graph and comparing it with the path generated by the medical language model under test to determine semantic coherence.

[0041] S5: Generate an evaluation report based on the similarity calculated in step S3 and the logical review results in step S4.

[0042] The evaluation report includes: a quantitative similarity score, details of all detected inference path anomalies, source information on inconsistent content in the knowledge graph, and targeted suggestions for model improvement.

[0043] The present invention also provides a medical language model evaluation device based on dynamic cost graph matching and inference path logic review, the device comprising the following modules:

[0044] The knowledge graph module provides a reference knowledge graph, which includes reference entity nodes and reference relationship edges.

[0045] The content parsing module receives the output content of the medical language model under test and parses it into a graph structure containing entity nodes and relation edges. It vectorizes the text information of the nodes in the knowledge graph and the graph under test, and uses graph neural networks to fuse the topological structure information of the graph for vector optimization.

[0046] The similarity calculation module is used to calculate the graph edit distance between the graph under test and the knowledge graph based on the graph edit distance algorithm and through the dynamically calculated node replacement cost. The graph edit distance value is converted into a similarity score and normalized. Then, the local similarity score and global similarity score between the graph under test and the knowledge graph are calculated through dynamic scoring and multi-round interaction mechanism.

[0047] The logic review module is used to perform logic review on the reasoning path in the graph under test. The logic review includes at least one of the following: using a depth-first search algorithm to detect logical loops, or detecting semantic coherence by calculating the similarity of the embedding vectors of adjacent nodes on the path.

[0048] The report generation module generates an evaluation report based on the calculation results from the similarity calculation module and the review results from the logical review module.

[0049] The advantages of this invention are:

[0050] 1. The dual-path design ensures that the present invention can seamlessly connect with various types of medical LLMs, regardless of their output capabilities. It can efficiently process advanced models with structured output capabilities, and is also compatible with traditional conversational or text-generating models, greatly expanding the application scope of the present invention.

[0051] 2. This invention goes beyond word matching (such as BLEU / ROUGE). It achieves in-depth medical semantic accuracy assessment by parsing the output into a graph structure and using a dynamic cost graph edit distance (GED) algorithm to calculate similarity. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating a medical language model evaluation method based on dynamic cost graph matching and inference path logic review provided by the present invention. Detailed Implementation

[0053] This specific implementation details the steps of a medical language model evaluation method and device based on dynamic cost graph matching and inference path logic review. This invention achieves automated, structured, and semantic evaluation of medical LLM output content through steps such as constructing a medical knowledge graph, constraining the LLM output structure, generating embedding vectors, calculating graph edit distance, dynamic scoring, and detecting inference link anomalies. (Refer to...) Figure 1 Specifically, it includes:

[0054] S1: Construct a knowledge graph containing reference entity nodes and reference relationship edges.

[0055] This method uses Google CloudVertexAIMedLM-RAPI to simulate 7 to 15 rounds of patient-doctor consultation dialogues, generating approximately 1,000 high-confidence medical dialogue data. If API authorization is not available, the Mistral-7B-Instruct-v0.2 medical-enhanced hypermodel, finely tuned on PubMedAbstracts and PMCFull Text, is used to generate simulated dialogue data.

[0056] From unstructured or semi-structured text data, it is necessary to automatically extract structured knowledge, namely "entity-relationship-entity" triples, using natural language processing (NLP) technology. This implementation uses at least one of MedSpaCy and cTAKES for knowledge extraction. For example, from the sentence "The patient has been taking nifedipine for a long time due to hypertension", this module can extract the triples (hypertension, receiving treatment, nifedipine).

[0057] To eliminate ambiguity in medical terminology, such as synonyms and abbreviations, and to ensure knowledge interoperability, all extracted entities must be mapped to an internationally recognized medical standard coding system. This includes all disease- and diagnosis-related entities, uniformly mapped to the International Classification of Diseases, 10th Revision (ICD-10). For example, the entity "hypertension" is standardized to a unique code I10. This ensures that regardless of whether the text uses "hypertension," "essential hypertension," or "hypertensive disease," it points to the same standardized node in the atlas. For broader clinical concepts, such as symptoms, examinations, treatments, medications, and anatomical locations, SNOMEDCT (Systematized Nomenclature of Medicine - Clinical Terms) is used for coding. SNOMEDCT is an extremely comprehensive and detailed multi-axis set of medical terms; for example, the symptom "fever" is standardized to SNOMEDCT concept ID 386661006.

[0058] The extracted 'entity-relationship-entity' triples are organized into a directed graph G with attributes. k =(V K E k A V A E In this context, entities are represented as nodes, and relationships are represented as directed edges, each with corresponding attributes.

[0059] Node set V k Each node v in the diagram represents a standardized medical entity;

[0060] The set of directed edges E k Each directed edge (u,v) represents a predefined relationship between entities u and v. Edge types include:

[0061] CAUSES: such as "high blood pressure → heart disease";

[0062] TREATS: such as "Aspirin → Headache";

[0063] HAS_SYMPTOM (symptoms): such as "common cold → fever";

[0064] DIAGNOSES_BY (Diagnostic Method): such as "Heart Disease → Electrocardiogram".

[0065] Node attribute A V Stored in a structured format (such as JSON), containing the following fields:

[0066] id: A unique node identifier;

[0067] name: The standard name of the entity, such as "hypertension";

[0068] type: The ontological type of the entity, such as "disease", "symptom", "drug", "examination";

[0069] code: Standardized encoding, such as "ICD-10:I10";

[0070] description: A textual description of the entity;

[0071] embedding: Reserved field, used to store the node embedding vectors generated later.

[0072] Edge attribute A E Include:

[0073] relation_type: Relation type, such as CAUSES (causes), TREATS (treatments), HAS_SYMPTOM (symptoms), DIAGNOSES_BY (diagnostic methods), etc.

[0074] confidence: The confidence score of this relationship, ranging from [0,1];

[0075] Source: Information tracing the origins of knowledge.

[0076] In a directed graph, nodes only receive messages from their incoming neighbors and only send messages to their outgoing neighbors, reflecting the unidirectional nature of medical relationships. For example, in the relationship "hypertension → heart disease," the heart disease node receives messages from the hypertension node.

[0077] Example of node attribute structure:

[0078] {

[0079] "id":"D123",

[0080] "type":"disease",

[0081] "name":"Hypertension",

[0082] Description: "Hypertension is a common chronic disease..."

[0083] "code":"ICD-10:I10",

[0084] "embedding":[0.12,-0.03,...]

[0085] }

[0086] To support subsequent logical reasoning and review, this knowledge graph, in addition to including the aforementioned basic relationships, specifically predefines Standard Evidential Paths during modeling. These paths are sequences of nodes and edges representing standard diagnostic and treatment procedures, extracted from authoritative clinical practice guidelines or extensive medical literature. The edges on these paths, in their edge attribute A... E Add an extra key-value pair, for example: "path_type":"standard_evidential_path". Example: For the diagnostic pathway of "acute myocardial infarction", a standard evidence-based pathway might include a series of nodes such as "chest pain symptoms" → "perform an electrocardiogram (ECG)" → "ST segment elevation on ECG" → "diagnosed as acute myocardial infarction". Pre-labeling this path in the knowledge graph provides a data foundation for the subsequent "key node missing detection" step.

[0087] S2: Receives the output of the medical language model under test, parses the output into a graph under test containing entity nodes and relation edges, vectorizes the text information of the nodes in the knowledge graph and the graph under test, and uses graph neural networks to fuse the topological structure information of the graph for vector optimization.

[0088] Methods for parsing the output include: constraining the medical language model under test to output as a triple array in JSON format, or, when the output is unstructured natural language, using a medical natural language processing toolkit for extraction and standardization.

[0089] Specifically, a JSON triple array, where each element is an object representing a knowledge triple, contains three properties: source (source entity), relation (relation), and target (target entity).

[0090] For example: [

[0092] {"source":"hypertension","target":"heart disease","relation":"pathogenicity","confidence":0.9},

[0093] {"source":"Aspirin","target":"Headache","relation":"Treatment"} ]

[0095] This method also supports other Semantic Web standard formats, such as Resource Description Frame (RDF) or Web Ontology Language (OWL), which are better compatible with existing knowledge representation protocols.

[0096] For this type of output, the system only needs to read the data and directly instantiate the graph G to be tested based on its structure. q The names of the nodes and edges, and the entities, will be used for subsequent standardization and embedding generation.

[0097] When the output of the medical language model under test is unstructured natural language text, such as a paragraph describing a patient's condition, it is processed through an on-demand NLP pipeline similar to that used in knowledge graph construction. Specifically, medical NLP tools such as MedSpaCy or cTAKES are used to extract entities and relations from the text. For example, for the input text "The patient's fever may be caused by the flu," the system will extract the triples (fever, symptoms, flu). MedSpaCy is a Python toolkit specifically designed for clinical text processing. This toolkit performs high-precision entity extraction using rule-based methods. For example, TargetRules can be defined to accurately identify known diseases, symptoms, or drug names. cTAKES (Clinical Text Analysis and Knowledge Extraction)... System is a more comprehensive platform tool based on the UIMA framework, providing a complete set of customizable NLP processing pipelines. cTAKES can not only identify medical entities, but also extract the relationships between entities, such as LocationOf (anatomical location relationship) and DegreeOf (degree relationship), through its RelationExtractorAggregate and other components.

[0098] The extracted entities undergo a standardization process identical to that of a knowledge graph, namely, querying standard coding systems such as ICD-10 and SNOMEDCT to obtain their normalized representations. The normalized entities are then used as nodes, and the extracted relationships as edges to construct the graph G to be tested. q .

[0099] Furthermore, in order to make mathematical quantitative comparisons of semantics and structure, the discrete knowledge graph and the graph structure data of the graph under test must be mapped into a unified, continuous high-dimensional vector space.

[0100] This implementation preferably employs a language model pre-trained on a large biomedical corpus; specifically, BioBERT is the preferred model. Compared to the original BERT trained on a general corpus or ClinicalBERT trained on clinical notes, BioBERT's pre-training data highly overlaps with the data sources of the knowledge graph of this invention. This gives BioBERT a natural advantage in understanding complex medical terminology and contextual relationships, enabling it to generate higher-quality initial semantic representations.

[0101] For any node v in the graph, its text attributes are concatenated into a text segment, which is then input into the BioBERT model to obtain an initial embedding vector e of fixed dimensions. node :

[0102] e node =BioBERT(T) node )

[0103] Where T node It is the text description of the node, e node It is the node embedding vector output by the BioBERT model.

[0104] The node attributes are concatenated with the node embedding vector to form the initial complete feature vector h before entering the graph neural network. node , represented as:

[0105] h node =[e node ,e type ,e code ,t confidence ]

[0106] Among them, e type ,e code ,t confidence These represent the node type, standard encoding, and confidence level, respectively.

[0107] Graph Attention Network (GAT) is used to fuse the topological structure information of the graph, and the initial complete feature vector h is then processed. node Optimization is performed to obtain the final embedding vector. The final embedding vector contains both the textual information of the node and its connectivity in the graph.

[0108] The update process of a GAT layer is as follows: For the input features h of all nodes... i That is, the output of the previous layer, or, for the first layer, the initial complete feature vector h. node A shared, learnable linear transformation, parameterized by the weight matrix W, is applied to obtain higher-level features; for each node i in the graph, the attention coefficient e between it and each of its neighboring nodes j is calculated. ij This coefficient measures the importance of the features of node j to node i, and is expressed as:

[0109] e ij =LeakyReLU(a T [Wh i ||Wh j ])

[0110] Among them, Wh i and Wh j These are the results of linearly transforming the feature vectors of nodes i and j using the shared weight matrix W, respectively. || denotes the vector concatenation operation, and a is a weight vector of a learnable single-layer feedforward neural network. T It is the transpose of a, and LeakyReLU is a non-linear activation function.

[0111] To ensure the comparability of attention coefficients across different nodes and to transform them into a probability distribution, the softmax function is used to normalize the attention coefficients of all neighboring nodes of node i, resulting in the normalized attention weight α. ij , represented as:

[0112]

[0113] N i Let k be the set of all neighboring nodes of node i, k be an index variable, and exp be the exponential function e. x The new feature vector h′ of node i i It is achieved by using normalized attention weights α ij The result, obtained by weighted summation of the transformed features of all its neighboring nodes and finally applying a nonlinear activation function σ, is expressed as:

[0114]

[0115] To make the learning process more stable and capture diverse neighborhood information from different representation subspaces, this invention employs a multi-head attention mechanism. This mechanism is implemented by executing K independent attention processes (called "heads") in parallel, each head using an independently learned parameter set (W). k ,a k This allows for the separate attention to different combinations of feature dimensions. The outputs of the K attention heads are aggregated using different strategies based on their position in the network layer:

[0116] In the intermediate layer, the feature vectors output by the K heads are concatenated. The principle behind this is that concatenation preserves all the information learned by each head from its different subspaces and passes it to the next layer, allowing the model to continue learning these diverse feature combinations at deeper levels. Its mathematical representation is:

[0117]

[0118] h′ i,concat This represents the new feature vector of node i obtained by concatenating the feature vectors output from the K heads in the intermediate layer. This involves concatenating vectors, and in the final output layer, the element-wise average (averaging) of the output vectors from the K heads is taken. The goal is to integrate the diverse information learned from multiple heads to generate a single embedding vector that matches the dimension of the input features. Its mathematical representation is:

[0119]

[0120] By stacking multiple layers of GAT networks, the generalized neighborhood information of nodes can be effectively captured. The mechanism lies in the fact that the output of the first-layer GAT aggregates the node's one-hop neighbor information; when this output is used as the input of the second-layer GAT, the node can indirectly aggregate the information of its two-hop neighbors. Therefore, a GAT model containing L layers enables the final embedding vector of each node to fuse structural and semantic information within its L-hop neighborhood, thereby generating a more global final node representation.

[0121] In scenarios where classification or comparison of the entire graph is required, a global embedding vector e representing the entire graph can be generated by performing a pooling operation on the final embedding vectors of all nodes in the graph, including but not limited to mean pooling or max pooling. graph , represented as:

[0122] e graph =Pooling({h′) v v∈V})

[0123] V here represents the set of all nodes in the graph, v is an index variable, and h′ v It is the final embedding vector of the node with index v. This embedding generation strategy lays the mathematical foundation for subsequent similarity calculation and logical review by organically combining the two core stages of BioBERT-based deep semantic understanding and GAT-based graph structure relationship reasoning.

[0124] S3: Based on the graph edit distance algorithm, the graph edit distance between the graph to be tested and the knowledge graph is calculated by dynamically calculating the node replacement cost. The graph edit distance value is converted into a similarity score and normalized. Then, the local similarity score and global similarity score between the graph to be tested and the knowledge graph are calculated through dynamic scoring and multi-round interaction mechanism.

[0125] For node v in the graph to be tested q And node v in the reference diagram k Its replacement cost C sub (v q ,v k It is composed of the following two weighted components:

[0126] C sub (v q ,v k ) = w sem *Cost sem (v q ,v k )+w type *Cost type (v q ,v k )

[0127] Among them, w sem and w type It is an adjustable weight hyperparameter that satisfies w sem +w type =1; First component Cost sem (v q v k The second component, Cost, is determined based on the semantic similarity between the node to be tested and the reference node. type (v q v k The determination is based on whether the node type of the node under test is consistent with the node type of the reference node.

[0128] The values ​​of these two weights can be set according to the specific evaluation focus. For example, in scenarios that emphasize the evaluation of the factual accuracy of the content, semantic similarity is more critical, and a value of w is preferred. semThe value is 0.7; however, in scenarios that emphasize assessing process compliance, node type matching is crucial, and it is preferable to set w to 0.7. type The value is 0.7. In a preferred embodiment, the optimal weight combination can be obtained by experimental tuning on a validation dataset pre-annotated by experts to maximize the correlation between the final score and the expert judgment.

[0129] The first component measures the semantic proximity between the node to be tested and the reference node. It is based on the final node embedding vector generated in step S2, which has incorporated structural information. and The calculation is performed. Specifically, the inverse of cosine similarity is used:

[0130]

[0131] When the embedding vectors of two nodes are very close in the vector space, that is, they are semantically highly similar, such as "myocardial infarction" and "heart attack", the surrogate value approaches 0; conversely, if they are semantically unrelated, the surrogate value approaches 1.

[0132] The second component introduces strong constraints from the medical ontology, penalizing node replacements that may be semantically related but are completely different in medical classification. Its calculation method is a penalty function:

[0133] Cost type (v q ,v k ) = 0iftype(v q ) = type(v k )

[0134] Cost type (v q ,v k ) = 1iftype(v q )≠type(v k )

[0135] For example, even if "diuretic" (type: drug) and "edema" (type: symptom) may co-occur in the text, resulting in a certain degree of similarity in their embedding vectors, the second component will impose a high cost due to their different node types. This effectively prevents the algorithm from producing medically absurd matches such as "replacing symptoms with drugs".

[0136] Furthermore, other editing operations have costs, including the cost of adding or deleting nodes (c). n , is a constant; the cost c for adding or deleting an edge. e , is a constant;

[0137] The graph edit distance (GED) algorithm comprehensively considers the dynamic node replacement cost (C) mentioned above. sub ), Node addition and deletion cost (c n ) and edge addition / deletion cost (c e ), calculate the image G to be tested q Convert to knowledge graph G k The minimum total cost required is the original graph edit distance value GED(G). q G k ).

[0138] To convert the calculated original graph edit distance (GED) values ​​into an intuitive similarity score between 0 and 1, where a higher score indicates better consistency, the scores are normalized and expressed as follows:

[0139] cost norm =|V q |+|E q |+|V k |+|E k |

[0140]

[0141] Among them, cost norm It is a normalization factor, usually related to the size of the two graphs, to ensure the relative comparability of the scores. In the formula, |V q |and|E q | These are the images to be tested, G. q The number of vertices and edges; |V k |and|E k |These are knowledge graphs G k The number of nodes and edges in the corresponding subgraph. Therefore, in this embodiment, this factor is defined as the total number of nodes and edges in both graphs, which represents the maximum possible cost of completely deleting one graph and reconstructing the other.

[0142] Furthermore, after each user input, the local similarity score for that round is calculated in real time. t The calculation method is as follows:

[0143] After each user input, the output graph G of the large language model under test is calculated in real time. q With knowledge graph subgraph G k Similarity score t score t The similarity scoring formula, score(G), defined above, is used. q G k The score is calculated as follows.t =score(G q G k );

[0144] The global similarity score is updated using a weighted average method.

[0145]

[0146] This represents the global score up to round t, where α∈[0,1] is the weight of the historical score. For example, α=0.5 means that the historical score and the current score have equal weight.

[0147] S4: Perform a logical review on the reasoning path in the graph to be tested. The logical review includes at least one of the following: using a depth-first search algorithm to detect logical loops, or detecting semantic coherence by calculating the similarity of the embedding vectors of adjacent nodes on the path.

[0148] Specifically, a depth-first search (DFS) algorithm is used to systematically traverse every node and edge in the reasoning path graph generated by the medical language model under test in order to identify whether circular arguments exist.

[0149] During Depth-First Search (DFS) traversal, a recursive stack is maintained to record all nodes on the current traversal path to detect loops. If, when visiting an adjacent node, it is found that the adjacent node already exists in the current recursive stack, the reasoning path is determined to contain a logical loop. For example, in the reasoning chain "hypertension → heart disease → hypertension", when the algorithm visits "hypertension" again from the "heart disease" node, which is already on the current path, this condition is met, thus detecting backtracking or a circular argument.

[0150] Furthermore, the similarity of the embedding vectors of adjacent nodes along the inference path is calculated. If the similarity is lower than a similarity threshold θ, it is determined to be a mid-path correction. θ is preferably 0.5. For example:

[0151] sim(e i ,e i+1 )<θ

[0152] e i ,e i+1 sim is the final embedding vector of two adjacent nodes on the inference path generated in step S2, sim is a similarity calculation function, and θ is a similarity threshold.

[0153] Furthermore, search for a reasonable path P from the starting point to the ending point in the knowledge graph. std , and the path P generated by the medical language model under test gen Compare.

[0154] If |P gen|<λ|P std If | λ < 0.5, or the key node set C is not covered, then there is a jump.

[0155] Judgment formula:

[0156]

[0157] S5: Generate an evaluation report based on the similarity calculated in step S3 and the logical review results in step S4.

[0158] Preferably, the evaluation report includes the following:

[0159] Overall score: Based on the combined results of multiple rounds of scoring.

[0160] Local scoring for each round: Displays the changes in scoring after each round of input.

[0161] Logical loop detection: Reports whether a logical loop exists.

[0162] Semantic coherence break detection: Reports whether there are semantic jumps.

[0163] Critical node missing detection: Determine whether there is a missing critical node in the report by whether the critical node set C is covered.

[0164] Key comparison nodes / relationships and tracing: List the nodes and relationships that do not match the knowledge graph and trace their origins.

[0165] Improvement suggestion: Generate suggestions based on the test results.

[0166] Furthermore,

[0167] Regarding logical loops: It is recommended to "optimize the reasoning chain to avoid repeated diagnosis";

[0168] Regarding the break in semantic coherence: it is recommended to "strengthen the consistency of continuous reasoning";

[0169] Regarding the missing key nodes: it is recommended to "complete the intermediate steps of the reasoning and cover the key nodes";

[0170] Low-scoring symptoms: It is recommended to "enhance the ability to reason about multiple symptoms together".

[0171] This implementation method realizes an intelligent evaluation system for medical LLM content that integrates structured, semantic, multi-turn interaction, automatic scoring and inference link anomaly detection, providing a solid technical foundation for the safe implementation of medical AI.

[0172] The present invention also provides a medical language model evaluation device based on dynamic cost graph matching and inference path logic review, the device comprising the following modules:

[0173] The knowledge graph module is used to provide a knowledge graph, which contains reference entity nodes and reference relationship edges.

[0174] The content parsing module receives the output content of the medical language model under test and parses it into a graph structure containing entity nodes and relation edges. It vectorizes the text information of the nodes in the knowledge graph and the graph under test, and uses graph neural networks to fuse the topological structure information of the graph for vector optimization.

[0175] The similarity calculation module is used to calculate the graph edit distance between the graph under test and the knowledge graph based on the graph edit distance algorithm and through the dynamically calculated node replacement cost. The graph edit distance value is converted into a similarity score and normalized. Then, the local similarity score and global similarity score between the graph under test and the knowledge graph are calculated through dynamic scoring and multi-round interaction mechanism.

[0176] The logic review module is used to perform logic review on the reasoning path in the graph under test. The logic review includes at least one of the following: using a depth-first search algorithm to detect logical loops, or detecting semantic coherence by calculating the similarity of the embedding vectors of adjacent nodes on the path.

[0177] The report generation module generates an evaluation report based on the calculation results from the similarity calculation module and the review results from the logical review module.

[0178] The medical language model evaluation device based on dynamic cost graph matching and inference path logic review proposed in this invention corresponds one-to-one with the aforementioned medical language model evaluation method based on dynamic cost graph matching and inference path logic review in terms of operation process and functional effect. Therefore, the structure and operation process of the medical language model evaluation device based on dynamic cost graph matching and inference path logic review will not be described again.

[0179] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can be implemented in the form of a computer program product on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Other embodiments may also be used. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating medical language models based on dynamic cost graph matching and inference path logic review, characterized in that, Includes the following steps: S1: Construct a knowledge graph containing reference entity nodes and reference relationship edges; S2: Receive the output content of the medical language model to be tested, and parse the output content into a graph to be tested containing entity nodes and relation edges. Vectorize the text information of the nodes of the knowledge graph and the graph to be tested, and use graph neural network to fuse the topological structure information of the graph for vector optimization. S3: Based on the graph edit distance algorithm, the graph edit distance between the graph to be tested and the knowledge graph is calculated by dynamically calculating the node replacement cost. The graph edit distance is converted into a similarity score and normalized. Then, the local similarity score and global similarity score between the graph to be tested and the knowledge graph are calculated by dynamic scoring and multi-round interaction mechanism. S4: Perform a logical review on the reasoning path in the graph to be tested. The logical review includes at least one of the following: using a depth-first search algorithm to detect logical loops, or detecting semantic coherence by calculating the similarity of the embedding vectors of adjacent nodes on the path. S5: Generate an evaluation report based on the similarity calculated in step S3 and the logical review results in step S4.

2. The medical language model evaluation method based on dynamic cost graph matching and inference path logic review according to claim 1, characterized in that, Step S1 includes: Simulated doctor-patient dialogue data is generated through a large language model, and based on this, medical NLP tools are used to automatically extract structured triples of "entity-relationship-entity". In order to eliminate the ambiguity of medical terminology and ensure the interoperability of knowledge, all extracted entities are mapped to the international standard medical terminology system. The structured triples are constructed into a directed attribute graph, where nodes represent standardized medical entities and edges represent predefined medical relationships; both nodes and edges are attached with structured attributes, including information such as encoding, type, confidence level, and knowledge traceability. The knowledge graph predefines standard evidence-based paths, which are sequences of nodes and edges representing standard diagnostic and treatment procedures mined from authoritative clinical practice guidelines or a large number of medical documents. The edges on these paths have an additional key-value pair added to their edge attributes to characterize the typical diagnostic and treatment procedures in authoritative clinical guidelines.

3. The medical language model evaluation method based on dynamic cost graph matching and inference path logic review according to claim 1, characterized in that, Step S2 includes: The medical language model under test is constrained to output a triple array in JSON format, or when the output is unstructured natural language, it is extracted and standardized using a medical natural language processing toolkit. An array of JSON triples, where each element is an object representing a knowledge triple. This object contains three properties: source (source entity), relation (relation), and target (target entity). When the medical big language model under test outputs unstructured natural language text, entities and relationships are extracted from the text through an instant NLP processing pipeline similar to that used when constructing the knowledge graph. The extracted entities are subjected to a standardization process that is completely consistent with the knowledge graph, namely, querying the standard coding system to obtain their standardized representation, using the standardized entities as nodes and the extracted relationships as edges to construct the graph to be tested; By using a pre-trained model, for any node in the graph, its text attributes are concatenated into a text, and this text is input into the pre-trained model to obtain a node embedding vector of fixed dimensions. The node attributes are concatenated with the node embedding vector to form the initial complete feature vector before entering the graph attention network; The graph attention network is used to fuse the topological information of the graph and optimize the initial complete feature vector to obtain a final embedding vector that includes the text information of the nodes and their connection relationships in the graph.

4. The medical language model evaluation method based on dynamic cost graph matching and inference path logic review according to claim 3, characterized in that, The optimization of the initial complete feature vector by using a graph attention network to fuse graph topological information includes: For the input features of each node, a linear transformation is performed using a shared weight matrix to enhance the node features, and the attention coefficient between nodes is calculated. The attention coefficients of all nodes are normalized using the softmax function. By using the normalized attention weights, the transformed features of all the neighboring nodes of a node are weighted and summed. Finally, a non-linear activation function is applied to obtain the new feature vector of the node. By using a multi-head attention mechanism to capture diverse information, the feature vectors output by multiple heads are concatenated in the intermediate layer, and the average value of the elements of the output vectors of multiple heads is taken in the output layer. This maximizes the preservation and transmission of information diversity within the model, ultimately producing a highly integrated and robust node representation. By stacking multiple layers of GAT networks, an L-layer model aggregates neighborhood features through L times, enabling the final representation of each node to capture and integrate the structural and semantic features of all higher-order neighborhoods within its L hops, thus achieving deep integration of information from local to global.

5. The medical language model evaluation method based on dynamic cost graph matching and inference path logic review according to claim 1, characterized in that, Step S3 includes: Based on the graph editing distance algorithm, the node replacement cost is calculated by weighting the first component and the second component. The first component is determined based on the semantic similarity between the node to be tested and the reference node, and the second component is determined based on whether the node type of the node to be tested is consistent with the node type of the reference node. Other editing operations have costs, including the cost of adding or deleting nodes and the cost of adding or deleting edges, which are all constants. By comprehensively considering the costs of node replacement, node addition / deletion, and edge addition / deletion, the minimum total cost required to convert the graph under test into a knowledge graph is calculated. The minimum total cost is the graph edit distance value. The calculated graph edit distance value is converted into a similarity score between 0 and 1, and then normalized. After each user input, the local similarity score for that round is calculated in real time, and the global similarity score is updated using a weighted average method.

6. The medical language model evaluation method based on dynamic cost graph matching and inference path logic review according to claim 1, characterized in that, In step S4: The method of detecting logical loops using a depth-first search algorithm includes: systematically traversing each node and edge in the reasoning path graph generated by the medical language model under test using a depth-first search algorithm to identify whether circular arguments exist; during the depth-first search traversal, a recursive stack recording all nodes on the current traversal path is maintained to detect loops; if when visiting an adjacent node, it is found that the adjacent node already exists in the current recursive stack, then the reasoning path is determined to contain a logical loop. The method of detecting semantic coherence by calculating the similarity of the embedding vectors of adjacent nodes on the reasoning path includes: calculating the similarity of the embedding vectors of adjacent nodes on the reasoning path; if the similarity is lower than the similarity threshold, it is determined to be a mid-course correction; searching for a reasonable path from the starting point to the ending point in the knowledge graph and comparing it with the path generated by the medical language model to be tested to determine semantic coherence.

7. The medical language model evaluation method based on dynamic cost graph matching and inference path logic review according to claim 1, characterized in that, The evaluation report mentioned in step S5 includes: a quantified similarity score, details of all detected inference path anomalies, source information on inconsistent content in the knowledge graph, and targeted model improvement suggestions.

8. A medical language model evaluation device based on dynamic cost graph matching and inference path logic review, characterized in that, include: The knowledge graph module is used to provide a reference knowledge graph, which includes reference entity nodes and reference relation edges; The content parsing module is used to receive the output content of the medical language model under test, parse it into a graph structure under test containing entity nodes and relation edges, vectorize the text information of the nodes of the knowledge graph and the graph under test, and use graph neural network to fuse the topological structure information of the graph for vector optimization. The similarity calculation module is used to calculate the graph edit distance between the graph to be tested and the knowledge graph based on the graph edit distance algorithm and through the dynamically calculated node replacement cost. The graph edit distance value is converted into a similarity score and normalized. Then, the local similarity score and global similarity score between the graph to be tested and the knowledge graph are calculated through dynamic scoring and multi-round interaction mechanism. The logic review module is used to perform a logic review on the reasoning path in the graph to be tested. The logic review includes at least one of the following: using a depth-first search algorithm to detect logical loops, or detecting semantic coherence by calculating the similarity of the embedding vectors of adjacent nodes on the path. The report generation module is used to generate an evaluation report based on the calculation results of the similarity calculation module and the review results of the logic review module.

Images