Medical term semantic analysis and evaluation method based on large language model

Abstract

The invention relates to a medical term semantic analysis and evaluation method based on a large language model. The method comprises the following steps: inputting a to-be-evaluated medical term pair into a computer system; the computer system retrieves definitions of the medical term pairs and the shortest semantic path of the definitions and the shortest semantic path of the definitions and the shortest semantic path from the UMLS, and supplemental definitions and external semantic paths are extracted from other authoritative external sources for terms missing information in the UMLS; constructing a unified prompt which contains the definition, the semantic path and a score of a generation set range and is based on a knowledge interpretation instruction, inputting the unified prompt into a large language model evaluation module, and generating a semantic similarity / correlation score and natural language interpretation; on the basis of the unified prompt, using supplementary information to verify whether explanation is compliant, has no information misdescription and has no illusion, allowing output only when the explanation, the information misdescription and the illusion are all satisfied, and otherwise, iteratively optimizing the prompt and regenerating; and outputting a structured result containing the semantic similarity / correlation score, the natural language interpretation, the knowledge reference and the supervision log.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically involving semantic understanding and semantic analysis in natural language processing, a key technology of artificial intelligence. In particular, it relates to a method for semantic analysis and evaluation of medical terminology based on a large language model. Background Technology

[0002] Semantic analysis and evaluation of medical terminology is a key technical support for ensuring the quality and efficiency of medical services. Its accuracy is directly related to patient safety, timely diagnosis, and the interoperability of medical systems.

[0003] In high-risk clinical scenarios such as emergency care and intensive care, accurately distinguishing relevant medical concepts, such as myocardial infarction versus angina, persistent cough versus pneumonia, can directly impact the timeliness and accuracy of treatment decisions, thereby saving patients' lives. At the medical information system level, this technology not only standardizes medical terminology in electronic health records (EHRs), eliminating data silos between different systems, but also provides semantic alignment support for medical cost management solutions such as Diagnosis Related Groups (DRG), ensuring the rationality of medical insurance reimbursement and hospital resource allocation. Biased semantic assessment can lead to clinical risks such as misdiagnosis, over-testing, and treatment delays, while also causing economic problems such as failed medical data integration and unnecessary medical expenses.

[0004] Current methods for semantic assessment of medical terminology are mainly divided into three categories, but all of them have significant technical limitations.

[0005] First, while ontology-driven approaches (such as those based on a unified medical language system) can provide a structured and interpretable knowledge framework, they are limited by the scope of predefined ontologies.

[0006] Secondly, corpus embedding methods (such as Word2Vec, BioBERT, and ClinicalBERT) learn semantic vectors through large-scale text, but lack transparency and interpretability, making it impossible to explain the scoring criteria to medical personnel. Furthermore, they have weak processing capabilities for low-frequency, out-of-vocabulary (OOV) terms, making it difficult to cope with the diverse terminology used in clinical practice.

[0007] Third, hybrid methods (such as BioWordVec and SapBERT) attempt to combine ontology knowledge with embedding models, but they often suffer from fragmented knowledge integration and insufficient fusion of multi-source information. While large language models (LLMs, such as GPT-3.5) that have emerged in recent years possess certain reasoning capabilities, they are prone to generating illusions (i.e., generating content without factual basis), lack reliable medical knowledge grounding, and have limited applicability in sensitive medical scenarios.

[0008] The shortcomings of existing technologies further exacerbate the risks in clinical applications and the operation of healthcare systems. Incomplete knowledge coverage means that some clinical terms (such as emerging disease names and colloquial expressions) cannot be effectively evaluated, creating blind spots in semantic understanding. The lack of interpretability makes it difficult for medical personnel to verify the reasonableness of evaluation results, reducing their trust in AI-assisted decision-making.

[0009] Studies show that patients and doctors prefer transparent AI systems to black-box models, even if the latter are more accurate. The illusions and semantic ambiguities inherent in LLM (Liquidity-Based Medical Imaging) can trigger a chain of errors, such as confusing the mechanisms of action of cefepime with bacitracin, or causing discrepancies in medical insurance reimbursement due to DRG grouping semantic alignment errors. This harms both patients' rights and the operational efficiency of the healthcare system.

[0010] With the acceleration of healthcare digitalization, the demand for semantic assessment technology for medical terminology has upgraded from accuracy to a three-dimensional upgrade encompassing accuracy, interpretability, and reliability. However, existing methods cannot simultaneously meet the requirements of comprehensive knowledge coverage, transparent reasoning, and reliable results, thus limiting their application in core scenarios such as clinical decision support and medical data standardization. Summary of the Invention

[0011] The main objective of this invention is to provide a method for semantic analysis and evaluation of medical terminology based on a large language model, thereby integrating multi-source medical knowledge, possessing an automated supervision and verification mechanism, and outputting structured and interpretable results.

[0012] Based on the first main aspect of the present invention, a method for semantic analysis and evaluation of medical terms based on a large language model is provided, which is implemented by the following steps:

[0013] Inputting the medical terms to be evaluated into the computer system ;

[0014] The computer system retrieves the medical terminology pairs from the Unified Medical Language System (UMLS). Definition , and the semantic shortest path between the two For terms lacking information in UMLS, supplementary definitions are extracted from other authoritative external sources. , and external semantic paths ;

[0015] A unified prompt, including the aforementioned definitions, semantic paths, and a defined range of scores, is constructed and input into the large language model evaluation module based on knowledge interpretation instructions to generate semantic similarity / relevance scores. and natural language interpretation;

[0016] Based on the unified prompt, supplementary information is used to verify whether the explanation is compliant, free of misrepresentation, and free of hallucination. Output is only allowed when all three conditions are met; otherwise, the prompt is iteratively optimized and regenerated.

[0017] The output includes structured results such as semantic similarity / relevance scores, natural language explanations, knowledge references, and supervision logs.

[0018] As a further preferred option, in the aforementioned method, the other authoritative external sources include one or a combination of UpToDate, MSD manual, and Baidu Health;

[0019] The semantic shortest path is retrieved from the UMLS knowledge graph, and the external semantic path is derived from the external knowledge graph. The maximum length of the semantic shortest path is set to 6. If the length exceeds 6, the term pair is deemed to be too weakly associated and discarded. All path information is converted into natural language.

[0020] As a further preferred embodiment, in the aforementioned method, the construction of a unified prompt that includes the definition, semantic path, and a score generated within a defined scope, and is based on knowledge explanation instructions, includes:

[0021] The basic information is categorized and integrated, and the terminology definitions retrieved from UMLS and the supplementary definitions extracted from authoritative external sources using the knowledge extraction tool LightRAG are organized into text paragraphs according to the first structured format.

[0022] The shortest path and external derivation path obtained from UMLS retrieval are converted into natural language, and the semantic types of terms and the relationships between terms are supplemented. The text is then sorted into text paragraphs according to the second structured format. The text paragraphs obtained from the two sorting processes are combined to form basic information text and stored in the computer system.

[0023] The core instruction is embedded into the basic information text. This instruction explicitly requires the generation of a semantic score of 1-10, and the explanation must refer to the terminology definition, supplementary definition, or shortest path or external derivation path, while simultaneously annotating the knowledge source and uncertainty.

[0024] Following the sentence structure specifications for adapting to large language model reasoning, the basic information text and core instructions are concatenated into a coherent text statement that adapts to large language model reasoning, and then used as a unified prompt.

[0025] As a further preferred embodiment, in the aforementioned method, the first structured format is: Term to be evaluated: [...], Corresponding definition: [...], Information source: [...];

[0026] The second structured format is: Term pair: [...], Path content: [...], Term semantic type: [...], Relationship between terms: [...].

[0027] As a further preferred embodiment, in the aforementioned method, the large language model evaluation module includes at least a scorer and a supervisor;

[0028] The rater is invoked to generate a rating, explanation, and reflection statement using a large language model; the reflection statement includes at least a text paragraph explaining the source of knowledge, the basis for the rating, and the uncertainty.

[0029] The supervisor uses the unified prompt as the authoritative reference and independently executes the verification process.

[0030] As a further preferred embodiment, in the aforementioned method, the steps of verifying and outputting the explanation based on the unified prompt and supplementary information include:

[0031] The verification is performed by the supervisor module in the large language model evaluation module. This supervisor module uses the term definitions and semantic paths contained in the unified prompts as supplementary information and authoritative references to perform three verifications on the natural language explanations generated by the scorer.

[0032] Among them, the compliance verification judgment explanation fully utilizes the above supplementary information; the non-information misrepresentation verification judgment explanation correctly cites the supplementary information without distortion or misuse; and the non-illusion verification judgment explanation does not contain any erroneous content not mentioned in the supplementary information.

[0033] The supervisor module encodes the validation results into a structured JSON object, which contains binary fields a, b, and c corresponding to the three validation results, and a final decision field permit. permit is set to true and interpretation is allowed only when a=true, b=false, and c=false; otherwise, the following iterative optimization is initiated:

[0034] By restating the task instructions or adjusting supplementary information to emphasize key points and optimize unified prompts, the task is resubmitted to the scorer to generate a new interpretation. The process continues until the interpretation passes verification or the predefined maximum number of retries is reached. The entire verification process and iterative optimization records are stored in the supervision log.

[0035] As a further preferred option, in the aforementioned method, when generating scores and explanations, the decoding parameters of LLM are set as follows: Temperature=1.0, Max Tokens=512, Top-p=0.7, Top-k=50;

[0036] Furthermore, each term pair generates three outputs, and the average value is taken as the final score to mitigate the variability of autoregressive decoding.

[0037] Based on the second main aspect of the present invention, a medical terminology semantic analysis and evaluation system based on a large language model is provided, including a knowledge retrieval and enrichment module, a large language model evaluation module, and a structured output generation module. These modules are sequentially connected and work together to achieve medical terminology semantic evaluation.

[0038] The knowledge retrieval and enrichment module is used to retrieve the definitions of the medical term pairs to be evaluated and the shortest semantic path between them from the Unified Medical Language System (UMLS). For terms with missing information in UMLS, the knowledge extraction tool LightRAG is used to extract supplementary definitions and external semantic paths from authoritative external sources, including UpToDate and the MSD manual. At the same time, all semantic paths are converted into natural language and the semantic types and relationships between terms are supplemented to form the basic information required for unified prompts.

[0039] The large language model evaluation module includes a scorer and a supervisor. The scorer calls the large language model and generates a semantic similarity / relevance score of 1-10 and a natural language explanation of the associated knowledge based on the unified prompts constructed based on the basic information. The supervisor uses the unified prompts as an authoritative reference to verify the explanations and output them, or iteratively optimizes the prompts and drives the scorer to regenerate.

[0040] The structured output generation module is used to integrate and output structured results that include the score, natural language explanation, knowledge reference, and supervision log.

[0041] Based on a third key aspect of the present invention, an electronic device is provided, characterized in that it comprises: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0042] The memory stores a computer program that, when executed by the processor, causes the processor to perform the aforementioned method for semantic analysis and evaluation of medical terms based on a large language model.

[0043] Based on a fourth key aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, characterized in that, when the program is executed, it implements the aforementioned method for semantic analysis and evaluation of medical terms based on a large language model.

[0044] Compared with existing technologies, the present invention addresses the pain point of incomplete knowledge coverage in traditional medical terminology semantic assessment methods by integrating multi-source authoritative knowledge. The present invention employs a dual-path retrieval strategy combining UMLS core knowledge with supplementary information from external authoritative sources. It extracts terminology definitions and semantic paths from UMLS, and then uses the LightRAG tool to extract supplementary information from authoritative medical sources such as UpToDate and the MSD manual. Simultaneously, the semantic paths are converted into natural language and limited to a maximum length of 6 (exceeding this length indicates weak association). This ensures both the authority and completeness of the knowledge, and by adapting structured processing to large language model reasoning, it enhances the assessment capability for rare and cross-domain medical terms.

[0045] Another key feature of this invention is its strong interpretability and structured output design, overcoming the limitations of existing black-box evaluation methods. The output of this invention integrates a semantic score of 1-10, natural language explanation, knowledge reference (clearly annotated definitions and path sources), and a supervision log (JSON format recording verification results). The explanation must strictly reference the retrieved terminology definitions or semantic paths, while the supervision log fully preserves the compliance, misrepresentation, and illusion verification processes. This structured design not only makes the scoring criteria clear at a glance but also supports manual auditing, enabling the evaluation results to be directly integrated into clinical workflows (such as electronic health record terminology standardization and Diagnosis Related Groups (DRG) classification). This represents a significant advancement in clinical adaptability that is difficult to achieve with existing methods.

[0046] This invention also effectively avoids the "illusion" problem common in Large Language Models (LLMs) through a lightweight supervision mechanism, balancing reliability and efficiency. The supervisor module of this invention uses unified prompts as the authoritative reference and performs three core checks on the explanations generated by the scorer (whether supplementary information is used, whether misstatement is used, and whether illusion exists). Output is only allowed if all three checks pass; otherwise, iterative generation is performed by optimizing prompts (the default maximum iteration is 2 times, using GLM4-9B as the supervisor). Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0048] Figure 1 The following is a flowchart illustrating the execution of a medical terminology semantic analysis and evaluation method based on a large language model in one embodiment of the present invention;

[0049] Figure 2A schematic diagram illustrating a medical terminology semantic assessment task, its clinical application, and possible results in one embodiment of the present invention is shown.

[0050] Figure 3 The overall architecture of the UMEval medical terminology semantic analysis and evaluation system is shown in one embodiment of the present invention. Detailed Implementation

[0051] The preferred embodiments of the present invention will be described in detail below to provide a clearer understanding of the purpose, features, and advantages of the invention. It should be understood that the following embodiments are not intended to limit the scope of the invention, but are merely illustrative of the essential spirit of the technical solution of the invention.

[0052] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known techniques associated with the invention may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0053] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0054] In the following embodiments, the technical terms may have the following meanings:

[0055] UMEval: This invention proposes a unified and interpretable framework for semantic analysis and evaluation of medical terms. By integrating multi-source biomedical knowledge, authoritative clinical resources, and a lightweight supervision mechanism, it outputs structured results that include semantic scores, natural language interpretations, knowledge references, and supervision logs, enabling accurate evaluation of the semantic similarity and relevance of medical terms.

[0056] The Unified Medical Language System (UMLS) is an internationally recognized system for integrating biomedical terminology. It provides the core knowledge source for this invention and is mainly used to retrieve the standard definitions, semantic types, and semantic shortest paths between medical term pairs to be evaluated. When terminology information is missing, it needs to be supplemented by external authoritative sources.

[0057] LightRAG: The knowledge extraction tool used in this invention has the core function of converting unstructured text (such as disease mechanisms and drug descriptions) from authoritative external sources such as UpToDate and MSD manuals into structured semantic units to supplement the missing medical terminology information in UMLS.

[0058] Large Language Model Evaluation Module: The core component responsible for generating semantic evaluation results in this invention, consisting of a scorer and a supervisor. The scorer calls the Large Language Model (LLM) to generate semantic scores and natural language explanations, while the supervisor verifies the compliance of the results based on unified prompts. The two work together to ensure the accuracy and interpretability of the evaluation results.

[0059] Unified prompt: The input large language model is a structured task carrier consisting of two parts: basic information and core instructions. The basic information includes term definitions (UMLS / external sources) and natural language semantic paths. The core instructions specify the scoring range (1-10 points) and interpretation requirements. The whole is strung together into a coherent text according to the sentence structure adapted to LLM inference.

[0060] Semantic Shortest Path: The shortest association path connecting the pairs of terms to be evaluated, retrieved from the UMLS knowledge graph. The maximum length is set to 6 (if it exceeds this, the term association is considered too weak and discarded). It needs to be converted into natural language and supplemented with the semantic type of the terms (e.g., drug-antibiotic, disease-infectious disease) and the relationship between the terms (e.g., mechanism of action-inhibition, etiology-caused by…) to provide association basis for semantic evaluation.

[0061] Pearson correlation coefficient: One of the statistical indicators used in this invention to evaluate model performance. Its core function is to measure the degree of linear consistency between the model's predicted semantic score and the expert-annotated true value. The value ranges from -1 (perfectly negative correlation) to 1 (perfectly positive correlation). The closer the value is to 1, the higher the consistency between the evaluation result and the expert's judgment.

[0062] Spearman rank correlation coefficient: One of the statistical metrics used in this invention to evaluate model performance. Based on the semantic scoring ranking results of term pairs, it measures the monotonic consistency between the model's predicted ranking and the expert-annotated ranking. It is robust to outliers and nonlinear relationships and is suitable for judging the global ranking accuracy of semantic evaluation.

[0063] Kendall-Tau correlation coefficient: One of the statistical metrics used in this invention to evaluate model performance. It quantifies the local order consistency of semantic scores by counting the number of consistent term pairs (the model and expert scores are ranked the same) and inconsistent term pairs (the rankings are different). It is more sensitive to changes in local rankings and is suitable for evaluation in high-resolution clinical scenarios.

[0064] Supervision Log: The output record of the supervisor module in this invention is stored in JSON format.

[0065] Semantic similarity: One of the core evaluation objects of this invention refers to the degree of similarity between two medical terms in terms of conceptual essence and core attributes (e.g., cefepime and bacitracin are both antibiotics and have high semantic similarity). The evaluation result is expressed as a score of 1-10, with higher scores indicating that the conceptual essence is closer.

[0066] Semantic relevance: One of the core evaluation objects of this invention refers to the degree of relevance between two medical terms in terms of application scenarios and association logic (such as the relevance between pneumonia and cefepime in terms of treatment and being treated). It is evaluated independently of semantic similarity and is also expressed on a scale of 1-10 to reflect the degree of relevance.

[0067] Hallucination (LLM generation): In this invention, the LLM generation error that needs to be avoided by the supervisor refers to the large language model generating content that does not exist in the unified prompt (including UMLS / external source knowledge) and has no authoritative basis (such as fabricating drug action mechanisms or creating terminological relationships). The supervisor uses hallucination checks to ensure that the interpretation does not contain such errors.

[0068] like Figure 1 As shown, in one embodiment of the present invention, a method for semantic analysis and evaluation of medical terms based on a large language model includes the following steps S110-S150:

[0069] S110, Input the medical terms to be evaluated into the computer system. ;

[0070] S120, the computer system retrieves the medical terminology pairs from the Unified Medical Language System (UMLS). Definition , and the semantic shortest path between the two For terms lacking information in UMLS, supplementary definitions are extracted from other authoritative external sources. , and external semantic paths ;

[0071] S130: Construct a score that includes the aforementioned definition, semantic path, and generated within a defined scope, and input it into the large language model evaluation module based on a unified prompt using knowledge interpretation instructions to generate a semantic similarity / relevance score. and natural language interpretation;

[0072] S140, based on the unified prompt, use supplementary information to verify whether the explanation is compliant, without misrepresentation, and without illusion. Output is allowed only when all three conditions are met; otherwise, iteratively optimize the prompt and regenerate it.

[0073] S150, output a structured result containing the semantic similarity / relevance score, natural language explanation, knowledge reference, and supervision log.

[0074] In one possible implementation, this invention proposes a unified and interpretable medical terminology semantic analysis and evaluation framework, UMEval, the implementation of which is described in detail below.

[0075] Given a pair of medical terms and Semantic evaluation aims to assess the degree of similarity or correlation between two entities. This task is crucial for ensuring clinical safety and diagnostic accuracy, and also supports various downstream medical artificial intelligence (AI) applications. Figure 2 As shown, a reliable assessment system can distinguish subtle disease categories, ensure consistency in terminology in medical records, and provide guidance for cost-sensitive decisions such as diagnosis-related group (DRG) classification and patient grouping.

[0076] However, the field still faces three long-standing challenges: (1) Ontologies such as the Unified Medical Language System (UMLS) often have problems such as incomplete coverage or outdated content; (2) Neural embedding models lack interpretability and transparency; (3) Large language models are prone to "illusions" (generating content without factual basis) when processing fuzzy inputs, and the output results are inconsistent.

[0077] To address the aforementioned limitations, this embodiment proposes UMEval—a unified and interpretable evaluation system composed of three tightly integrated modules, such as... Figure 3 As shown.

[0078] The first module is the knowledge retrieval and enrichment module ( Figure 3 (b) It retrieves semantic definitions and conceptual paths from the Unified Medical Language System (UMLS). For medical terms that are not adequately defined or not included in UMLS, the system supplements the information through a knowledge enhancement submodule, which extracts definitions from authoritative sources such as UpToDate and MSD Manuals. This heterogeneous information is processed into a unified representation, thereby enriching information for both registered and out-of-vocabulary (OOV) terms.

[0079] The second module is the evaluation module based on a large language model. Figure 3c) This module assesses semantic similarity using controlled prompting. The rater generates a similarity score, text explanation, and feedback metadata based on enriched knowledge and task instructions. A supervisor then checks whether the output matches the prompt content and whether the reasoning process conforms to the provided definitions. If inconsistencies are found, the module can provide corrective feedback to optimize the results.

[0080] The third module is the interpretable output generation module. Figure 3 d) It formats the results into structured output, including similarity scores, explanations, supporting knowledge references, and supervision logs. This information supports traceability and auditability and facilitates the integration of results into clinical workflows. The output generated by UMEval is expected to support various medical artificial intelligence applications. Figure 3 e).

[0081] exist Figure 3 The UMEval method comprises four stages: terminology input, knowledge retrieval and enrichment, large language model-based evaluation, and interpretable output generation. Medical terms retrieved from the Unified Medical Language System (UMLS) are marked in blue, while out-of-vocabulary (OOV) terms are marked in orange. The large language model-based rater generates similarity scores and interpretive inferences; the supervisor validates the results according to instructions to ensure factual consistency. The final output includes interpretable evaluation results, as well as traceable explanations and knowledge references.

[0082] The implementation process of the knowledge retrieval and enrichment module is described in the following embodiment.

[0083] Existing medical ontologies (such as the Unified Medical Language System UMLS) have limited coverage and often omit emerging, rare, or colloquial terms. This poses a challenge to evaluation models based on large language models—insufficient contextual information often leads to inaccurate outputs or produces "illusions" (generating content without factual basis).

[0084] As an authoritative biomedical ontology, the Unified Medical Language System (UMLS) provides a systematic classification, definition, and entity relationships for terminology. In this embodiment, the definition of each term is first extracted from UMLS. In order to clarify its semantic connotation.

[0085] To construct a structured semantic representation of term pairs, this embodiment further retrieves term pairs from a UMLS-based knowledge graph. The shortest path between At the same time, supplement the semantic types of the terms involved in this path. and its relationship with other terms The path information is then converted into natural language to improve readability and interpretability. In this embodiment, the maximum path length is set to 6; if the shortest path exceeds this length, the term pair is considered to have a weak association and is discarded. This structured extraction method ensures both the integrity and coherence of the information and lays the foundation for downstream reasoning.

[0086] The following embodiment illustrates knowledge enrichment based on an authority network.

[0087] To fill information gaps in UMLS (such as missing definitions or semantic relationships), this embodiment introduces an external knowledge enrichment mechanism. This mechanism consists of two parts: first, selecting authoritative medical websites as external knowledge sources; and second, using specialized knowledge extraction tools to convert unstructured content into structured semantic information.

[0088] In most embodiments, the selection of authoritative medical websites is based on domain credibility and editorial quality, including UpToDate, MSD Manuals, and Baidu Health. When a certain term... Or a certain term When corresponding information is lacking in UMLS, the system initiates a hierarchical query process, sequentially retrieving these selected sources and collecting relevant descriptive or related content. This design ensures that the retrieved information is based on reliable medical resources, thereby improving the factual accuracy and clinical relevance of the results.

[0089] The collected online content is then processed by a knowledge extraction tool, converting unstructured text into structured semantic units such as definitions, entity references, and inter-term relationships. In this embodiment, LightRAG is used as the core extraction tool within the UMEval framework. LightRAG combines retrieval-augmented generation with structured output capabilities, enabling the extraction of term definitions. And construct an external knowledge graph If necessary, the shortest path representation can be derived from this external knowledge graph. .

[0090] If neither UMLS nor external sources can provide usable results, the system will directly use term pairs as prompt words to input into the large language model and rely on the internal knowledge pre-trained by the model to perform reasoning.

[0091] The unified prompt word construction mechanism is illustrated in the following embodiment.

[0092] In this embodiment, regardless of whether the information originates from UMLS or an external network, all retrieved knowledge is integrated into unified prompt terms. The constructed prompt terms include definitions, semantic types, and relational paths, and these components are formatted to improve compatibility with large language model inference. This mechanism ensures that the model operates based on a robust knowledge base and strictly adheres to clinically relevant semantics.

[0093] In one possible implementation, the present invention uses the term pair “cefepime” and “bacitracin” as an example to demonstrate the results of external knowledge retrieval, including the definitions of both and the shortest path information obtained from the UMLS knowledge graph.

[0094] Example: External Knowledge Retrieval for Terminology Pairs Concept 1: Cefepime Definition: A semi-synthetic, broad-spectrum fourth-generation cephalosporin with antibacterial activity. Cefepime binds to and inactivates penicillin-binding proteins (PBPs) on the inner membrane of bacterial cell walls. Penicillin-binding proteins are enzymes involved in the terminal stages of bacterial cell wall assembly and cell wall remodeling during bacterial growth and division. Inactivation of penicillin-binding proteins interferes with the cross-linking of peptidoglycan chains—which is essential for maintaining the strength and rigidity of the bacterial cell wall. This leads to weakening of the bacterial cell wall, which in turn triggers cell lysis.

[0095] Concept 2: Bacitracin Definition: An industry dedicated to creating products and procedures aimed at improving appearance and aesthetic appeal.

[0096] Path information: The shortest path between two terms in the knowledge graph. From cefepime to bacitracin via relation "AQ" (relation description: allowed qualifiers, relation attribute description: no description). From "standard features" to "beauty culture" via relation "QB" (relation description: can be qualified by..., relation attribute description: no description).

[0097] This example demonstrates the external knowledge retrieval results for the medical term pair (cefepime-bacitracin). The core of the results includes two key types of information: "term definition" and "knowledge graph semantic path". It is an output instance of the knowledge retrieval and enrichment module in the UMEval framework, providing a foundation for subsequent construction of unified prompts and generation of semantic evaluation results.

[0098] Among them, the definition of Concept 1 (cefepime) is consistent with its actual medical attributes (fourth-generation cephalosporin), clearly elucidates the antibacterial mechanism (binding to PBPs → interfering with peptidoglycan cross-linking → bacterial lysis), and the information is accurate and has clinical reference value.

[0099] The definition of Concept 2 (bacitracin) is clearly contradictory: Bacitracin is actually a polypeptide antibiotic, mainly used to treat local bacterial infections such as those on the skin, while it is defined in the example as "beauty-related industries", which is presumably a data error in the UMLS or external retrieval process.

[0100] The following examples illustrate in detail the method for evaluating the semantic similarity of medical terms based on a large language model.

[0101] To improve the consistency, reliability, and interpretability of semantic similarity assessment for medical terminology, this invention proposes a structured assessment module that combines instruction design with automated supervision. For example... Figure 3 As shown in Figure c, this module consists of three functionally independent but tightly integrated components: a prompt, a rater, and a supervisor. This architecture ensures that the Large Language Model (LLM) can not only generate richly informative scores, but also that its explanations are based on domain knowledge and possess verifiable traceability. Figure 3 c also provides a specific example of the evaluation process, in which the semantic similarity between cefepime and bacitracin is evaluated using structured cue words, and then verified by a supervisor.

[0102] The assessment process begins with constructing semantically rich and structurally constrained cue words. Each cue word contains a pair of medical terms that serve as the assessment target, denoted as . Each term is accompanied by its own definition—derived from the Unified Medical Language System (UMLS) and supplemented by external knowledge retrieval when necessary. If a shortest semantic path connecting two terms exists, the prompt word will also include this path to provide contextual information. To improve readability, this path is converted to natural language. Furthermore, the prompt word embeds explicit instructions specifying the expected output: a numerical similarity score and an explanation of the basis for the provided knowledge. This design ensures that the large language model operates under clear expectations and that the downstream inference process remains consistent with the input semantics.

[0103] The following embodiment illustrates a scorer (output generation) with explanation and self-monitoring.

[0104] After receiving structured prompts, the scorer invokes a large language model to generate a multi-part response tailored to the evaluation. First, the large language model generates a real-valued similarity score. This score reflects the input terms and The estimated semantic closeness between them.

[0105] After generating the scores, the large language model provides a concise natural language explanation—based on the content of the prompt words, referencing terminology definitions and semantic relationships. To enhance transparency and self-regulation, the large language model also outputs a brief reflection, indicating the sources of knowledge used, the core basis for the scoring, and potential uncertainties. The complete output is then fed into a validation process for quality control.

[0106] The following embodiment illustrates a supervisor with automated verification and regeneration.

[0107] To mitigate common errors (such as omission of key information, factual distortion, or the "illusion" of generating unfounded content), this embodiment proposes a lightweight and practical supervisor module. This component acts as a post-processing validator, using the original prompt content as an authoritative reference to independently check the output generated by the scorer module. Specifically, it focuses on two common failure scenarios in the interpretation and generation process of large language models: (1) ignoring knowledge components provided in the prompts (such as terminology definitions or graph paths); and (2) misusing, misrepresenting, or fabricating unfounded information.

[0108] In one possible implementation, the supervisor receives three inputs: the original prompt words (including definition and path information), a similarity score generated by the scorer, and a natural language explanation. The supervisor then evaluates the explanation based on three core criteria:

[0109] (a) Compliance check: Explain whether the supplementary information provided in the prompt words (such as definitions, semantic paths) has been fully utilized;

[0110] (b) Misrepresentation check: Explain whether the above information is correctly cited and whether there is any distortion or misuse;

[0111] (c) Hallucination check: Explain whether there is no “hallucination” content or unverifiable information.

[0112] The validation result is encoded as a structured JSON object containing four binary fields (a, b, c, permit) and a human-readable feedback field. The permit field is automatically set to true only if the boolean expression "a = true, b = false, and c = false" is true. This logical condition ensures that validation passes only if the interpretation faithfully and accurately uses the input information, without any misrepresentation or illusion.

[0113] If the verification result is permit=false, the system will initiate an iterative regeneration mechanism: in each round, the task instructions are rephrased or the key optimization prompts are adjusted, and then resubmitted to the scorer to generate a new interpretation. This cycle continues until the interpretation passes all verification checks or reaches the predefined maximum number of retries.

[0114] The introduction of this self-regulating loop not only reduces model "illusion" but also improves the system's robustness to input differences and semantic ambiguity. The core advantage of the supervision mechanism lies in its ability to achieve structured and automated quality assurance through symbolic standards, providing a traceable and auditable verification process, thereby enhancing the reliability and transparency of the evaluation results. The final output includes both the semantic evaluation results and accompanying verification metadata—both of which enhance user trust and support human-machine collaborative auditing when needed. After the output passes verification, it is marked as "permitted" and passed to the output generation module (see...). Figure 3 (d) The complete assessment results (including scores, explanations and verification metadata) will be retained to ensure transparency and to support human auditing when necessary.

[0115] The generation of structured interpretable output is illustrated in the following embodiments.

[0116] In high-risk environments such as healthcare, numerical output alone is far from sufficient. Users require transparent and interpretable results—clearly linking semantic scores to the underlying medical evidence. To enhance the transparency and interpretability of the scoring results, this implementation of UMEval simultaneously outputs structured natural language explanations, as well as explicit citations and verifications of knowledge sources. This module aims to fill the logical gap between the generated scores and the underlying knowledge, clarifying not only what the semantic evaluation results are but also why they are derived in this way.

[0117] By encapsulating assessment results in an interpretable and auditable format, UMEval supports both manual review and automated integration into clinical workflows. This final step ensures that semantic assessment is not only technically sound but also practical and credible. Figure 3 (d) illustrates the role of this module in the overall approach.

[0118] To ensure standardized and resolvable interpretation, the standard output format is defined as follows in this embodiment:

[0119] System Instructions: As a professional evaluator, you are responsible for verifying whether the explanations provided by the language model correctly utilize the input prompt materials (such as definitions and knowledge graph paths). You must evaluate the explanations and return a structured JSON object containing the following fields:

[0120] 'a': true / false (Compliance check: Does the explanation appropriately utilize the supplementary material in the prompt?)

[0121] 'b': true / false (Misrepresentation check: Explain whether any information in the prompt word has been misused or misrepresented?)

[0122] 'c': true / false (Illusion check: Does it introduce an error message not included in the prompt word?)

[0123] 'permit': true / false (Final decision: true is only set when a=true, b=false, and c=false)

[0124] 'feedback': "..." (A concise explanation of the evaluation results, including optimization suggestions if necessary)

[0125] In the above implementation process, only valid JSON objects should be returned, and no natural language content or other Markdown formats should be added outside of the JSON.

[0126] The "feedback" field should clearly and concisely explain the basis for the judgment of fields a, b, and c.

[0127] If any input component (such as definition, graph path) is missing or marked as "nan", but the explanation is supplemented by reasonable use of valid domain knowledge and no inaccurate information is introduced, it should still be judged as compliant use (a=true, b=false).

[0128] Example output: {"a": true, "b": false, "c": false, "permit": true, "feedback": "The interpretation is valid and accurately references the provided definition and path information."}

[0129] Output format: <Score: 'x.xx', Knowledge Reference: "...", Explanation: "...", Supervision Log: {...}>

[0130] The output consists of several components: a score representing the similarity or relevance of term pairs (ranging from 1 to 10); an explanation that is a well-reasoned statement generated based on the provided input knowledge; knowledge references that clearly present the external definitions and semantic path information used by the model; and a supervision log, a structured JSON verification result provided by the supervisor module, which clearly records the rationality and correctness of the explanation generation process. This combined output clearly describes the semantic relationships between terms, possesses strong readability and interpretability, and is therefore easy for manual review or direct use in downstream clinical applications.

[0131] Overall, in this embodiment, UMEval is designed around a "tightly integrated architecture," with each module addressing a specific pain point in semantic evaluation—namely, limited coverage, unstable reasoning, and insufficient interpretability. These modules collectively constitute a coherent and domain-adaptive approach to achieve robust understanding of medical terminology.

[0132] In one of the embodiments, an experiment is conducted to verify the method of the present invention.

[0133] To verify the effectiveness of the proposed method, this invention uses a publicly available reference dataset for semantic relevance and similarity of medical terms (Guo, Z., Xia, L., Yu, Y., Ao, T., Huang, C.: LightRAG: Simpleand Fast Retrieval).

[0134] Augmented Generation. Preprint at https: / / arxiv.org / abs / 2410.05779(2024)). This dataset contains six subsets, each consisting of medical concept pairs annotated by medical professionals such as medical coders, resident physicians, and clinicians.

[0135] The details of the 6 subsets are as follows:

[0136] MayoSRS: Contains 101 pairs of medical concepts, with semantic relevance scores manually annotated.

[0137] MiniMayoSRS: A smaller subset selected from MayoSRS, containing 29 pairs of medical concepts, with high inter-rater consistency.

[0138] UMNSRS-Sim: Contains 566 pairs of Unified Medical Language System (UMLS) concepts, using a continuous scale to label semantic similarity scores.

[0139] UMNSRS-Rel: Contains 588 pairs of UMLS concepts, and also uses a continuous scale to label semantic relevance scores.

[0140] UMNSRS-Sim(F) and UMNSRS-Rel(F): These are filtered versions of the original UMNSRS-Sim and UMNSRS-Rel datasets, respectively. Specifically, samples and concept pairs not found in the clinical corpus, biomedical corpus, or general English corpus were removed, resulting in 449 and 458 concept pairs remaining, respectively.

[0141] To comprehensively and rigorously evaluate the consistency between the model's predicted semantic scores and the expert-annotated ground truth, this embodiment employs three mature correlation-based statistical metrics (Pearson correlation coefficient). Spearman rank correlation coefficient Kendall-Tau correlation coefficient This approach combines the relative improvement rate (RI) metric for benchmark comparison with the aggregated correlation index (ACI) for comprehensive performance summarization. The rationale for selecting these metrics is that semantic similarity and relevance assessment in the medical field has multi-dimensional characteristics, requiring multiple metrics to cover different assessment perspectives.

[0142] Specifically, the three correlation coefficients capture complementary dimensions of consistency:

[0143] Pearson correlation coefficient: assesses the degree of linear consistency between predicted and reference scores;

[0144] Spearman rank correlation coefficient: reflects the preservation of the global sorting order;

[0145] Kendall-Tau correlation coefficient: emphasizes local order consistency.

[0146] Since semantic judgments can vary in numerical magnitude, sorting order, and pairwise comparisons, using these three metrics simultaneously creates a robust and discriminative evaluation framework. The RI metric quantifies the model's performance change relative to a benchmark model, while ACI provides a consistent representation of the model's consistency across different datasets and different relevance dimensions. This metric design meticulously characterizes model performance while considering both absolute and relative dimensions of semantic consistency, ensuring fair evaluation across scenarios with varying annotation styles and data distributions—a crucial prerequisite for deploying models in clinically sensitive settings.

[0147] Pearson correlation coefficient The Pearson correlation coefficient measures the linear correlation between two continuous variables. In this embodiment, the application of the Pearson correlation coefficient includes the following steps.

[0148] Given Grouped observations The calculation formula is as follows:

[0149]

[0150] in, and They are respectively and The sample mean. The value ranges from -1 (perfectly negative correlation) to 1 (perfectly positive correlation). The Pearson correlation coefficient has particular reference value when evaluating the absolute accuracy and proportional consistency between predicted scores and human annotations; in semantic similarity assessment, expert scores are often treated as continuous true values, therefore... It is a core indicator for measuring global numerical consistency.

[0151] Spearman Rank Correlation Coefficient The monotonicity of variables is evaluated based on their sorting order. In this embodiment, let... and They are respectively and The ranking is calculated using the following formula:

[0152]

[0153] This coefficient is robust to outliers and nonlinear relationships, making it particularly suitable for scenarios where similarity is more important than precise numerical consistency. In medical terminology evaluation, different annotators may only have an ordinal understanding of semantic closeness; in such cases… It can effectively reflect the global ranking accuracy of the model.

[0154] Kendall-Tau correlation coefficient Local order consistency is quantified based on the number of consistent and inconsistent pairs. The calculation formula is as follows:

[0155]

[0156] in, and These represent the number of consistent pairs and inconsistent pairs, respectively. and These represent the number of items that are tied (have the same rank) in the two variables, respectively. compared to, It is more sensitive to changes in local ordering and is an important tool for assessing the stability of the micro-semantic ordering of prediction results—which is especially crucial in high-resolution clinical decision support tasks.

[0157] To evaluate the performance improvement of UMEval relative to the baseline model, this embodiment also introduces the Relative Improvement Rate (RI) metric. RI is used to quantify the proportional performance gain of the target model relative to the reference model, providing a scale-independent measure of improvement, and its definition is:

[0158]

[0159] in, and These represent the evaluation scores of UMEval and the benchmark model, respectively. A positive RI value indicates that UMEval outperforms the benchmark model, while a negative value indicates a performance decline. By combining... , , The three correlation coefficients, RI, can comprehensively measure the relative performance of the model across different datasets and evaluation dimensions.

[0160] To further support consistent comparisons across different datasets and relevant dimensions, this embodiment also introduces the Aggregate Relevance Index (ACI) as a unified metric for model performance. ACI is the arithmetic mean of the three correlation coefficients across all datasets, and its calculation formula is as follows:

[0161]

[0162] Where K is the number of datasets, , , They represent the first Three correlation coefficients across the dataset. ACI can concisely summarize the overall consistency between the model and human annotations.

[0163] The above five metrics together constitute a comprehensive evaluation framework. This multi-perspective design enables both fine-grained and global performance evaluation of the model in semantic similarity and relevance tasks.

[0164] Experiments show that the UMEval system of this invention outperforms 13 representative baseline methods on six medical terminology semantic evaluation benchmark datasets (MayoSRS, MiniMayoSRS, UMNSRS-Sim, UMNSRS-Rel, UMNSRS-Sim (F), UMNSRS-Rel (F)), including ontology-based methods (such as Shortest-Path), corpus-based methods (such as SapBERT, BioLORD), and general / medical-specific large language models (such as GPT-3.5, GPT-4.1, Baichuan-M1, DeepSeek-V3, etc.).

[0165] In terms of core metrics, UMEval achieved the highest consistency with expert physician scores of 0.88, and its Kendall-Tau correlation coefficient (a measure of local order consistency) was 15.49% higher than the baseline method. Furthermore, its convergent correlation index (ACI, which is the average of Pearson, Spearman, and Kendall-Tau coefficients across all datasets) ranked first among all comparative models. Even when facing advanced models like GPT-4.1, UMEval still demonstrated superior stability in fine-grained semantic evaluation tasks.

[0166] For example, on the UMNSRS-Sim (566 pairs of UMLS concept similarity assessment) and UMNSRS-Sim (F) (449 pairs of filtered concepts) datasets, the Kendall-Tau coefficients are improved by 15.54% and 8.76% respectively compared to the base model DeepSeek-V3, which fully verifies the role of multi-source knowledge fusion and supervision mechanism in improving the accuracy of semantic ranking.

[0167] The performance differences between different types of baseline methods further highlight the advantages of UMEval.

[0168] For example, ontology base methods (such as Shortest-Path) suffer from incomplete coverage due to their reliance on predefined UMLS paths, resulting in an ACI of only 0.3482, which is far lower than UMEval's 0.7423.

[0169] Corpus-based methods (such as SapBERT and BioLORD) have improved semantic capture capabilities through embedding techniques (BioLORD has an ACI of 0.5010), but they lack contextual reasoning and interpretability, and are difficult to handle rare or out-of-vocabulary terms.

[0170] While general-purpose and medical-specific LLMs (such as GPT-4.1 with an ACI of 0.7348 and Baichuan-M1 with an ACI of 0.6892) possess strong reasoning abilities, they are prone to producing "hallucinations".

[0171] UMEval effectively compensates for the aforementioned shortcomings by using multi-source knowledge retrieval (external authoritative sources such as UMLS+UpToDate) and lightweight supervised verification (GLM4-9B as the supervisor, with a maximum of 2 iterations). On the UMNSRS-Rel (F) (458 concept pairs) and UMNSRS-Sim (F) datasets, which filter out rare concept pairs, its Pearson, Spearman, and Kendall-Tau coefficients all show a steady increase, indicating that knowledge enhancement is more effective in clean semantic environments. The improvement is only moderate on small datasets (such as MayoSRS, 101 concept pairs), as the limited sample size and narrow semantic range prevent the full realization of the advantages of knowledge integration.

[0172] The technical terms, principles, or means related to the technical solutions of the present invention mentioned in the above embodiments, which are not described in detail above, are all well-known technologies or common practices that are known to those skilled in the art.

[0173] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A medical term semantic analysis and evaluation method based on a large language model, characterized in that, The implementation is achieved by the following steps: Inputting medical terms to be evaluated into a computer system ; The computer system retrieves definitions of the medical term pair from the Unified Medical Language System (UMLS) , , and their semantic shortest paths , supplemental definitions from other authoritative external sources for terms missing information in the UMLS , and external semantic paths ; Constructing a unified prompt containing the definition, semantic path, and score of the generated setting range, and based on the knowledge explanation instruction, inputting a large language model evaluation module to generate a semantic similarity / relevance score and natural language explanation; Based on the unified prompt, the supplementary information is used to verify whether the interpretation is compliant, has no information misstatement, and has no illusion, and only when all three conditions are met, the output is allowed, otherwise the prompt is iteratively optimized and regenerated; The structured results include the semantic similarity / correlation score, natural language interpretation, knowledge reference, and supervision log.

2. The medical term semantic analysis and evaluation method based on a large language model according to claim 1, characterized in that, The other authoritative external sources include one or a combination of UpToDate, MSD Manual, and Baidu Health; The semantic shortest path is retrieved from the UMLS knowledge graph, the external semantic path is derived from the external knowledge graph, and the maximum length of the semantic shortest path is set to 6, and if it exceeds, it is determined that the term pair is weakly related and discarded, and the path information is converted into natural language. 3.The medical term semantic analysis and evaluation method based on a large language model according to claim 1, wherein, The unified prompt includes the following steps: Integrate the basic information by classification, the term definition retrieved from UMLS, and the supplementary definition extracted from the authoritative external sources by the knowledge extraction tool LightRAG, and organize them into text paragraphs according to the first structured format; Convert the shortest path retrieved from UMLS and the external derived path into natural language, supplement the semantic type of the term and the relationship between the terms, and organize them into text paragraphs according to the second structured format; the text paragraphs obtained by organizing the two parts constitute the basic information text and are stored in the computer system; Embed the core instructions into the basic information text, which explicitly require to generate a semantic score of 1-10 points, and the interpretation needs to reference the term definition, supplementary definition, or shortest path, external derived path, and simultaneously mark the knowledge source and uncertainty; According to the sentence specification suitable for large language model reasoning, concatenate the basic information text and core instructions into coherent text sentences suitable for large language model reasoning, and use them as unified prompts.

4. The medical term semantic analysis and evaluation method based on a large language model according to claim 3, characterized in that, The first structured format is: term to be evaluated: [...], corresponding definition: [...], information source: [...]; The second structured format is: term pair: [...], path content: [...], semantic type of term: [...], relationship between terms: [...].

5. The medical term semantic analysis and evaluation method based on a large language model according to claim 1 or 3, characterized in that, The large language model evaluation module at least includes a scorer and a supervisor; The scorer is used to generate scores, interpretations, and reflection statements using a large language model; the reflection statement at least includes text paragraphs explaining the knowledge source, score basis, and uncertainty; The supervisor independently performs the verification process with the unified prompt as the authoritative reference.

6. The medical term semantic analysis and evaluation method based on a large language model according to claim 5, characterized in that, The steps of verifying the interpretation and outputting based on the unified prompt include: The verification is performed by the supervisor module in the large language model evaluation module, which uses the term definition and semantic path contained in the unified prompt as supplementary information and authoritative reference to carry out three verifications on the natural language interpretation generated by the scorer; Among them, the compliance verification judges whether the interpretation sufficiently utilizes the above-mentioned supplementary information, the no information misstatement verification judges whether the interpretation correctly references the supplementary information without distortion or misuse, and the no illusion verification judges whether the interpretation contains incorrect content not mentioned in the supplementary information; The supervisor module encodes the verification result as a structured JSON object containing binary fields a, b, c corresponding to the three verification results respectively and a final decision field permit, which is set to true and allows the interpretation output only when a=true, b=false, c=false, otherwise an iterative optimization is initiated as follows: The unified prompt is optimized by re-emphasizing the task instructions or adjusting the supplementary information, resubmitted to the scorer to generate a new interpretation, and the cycle continues until the interpretation passes the verification or reaches the predefined maximum number of retries, and the entire verification process and iterative optimization records are stored in the supervision log.

7. The medical term semantic analysis and evaluation method based on a large language model according to claim 6, characterized in that, When generating the score and interpretation, the decoding parameters of the LLM are set as: Temperature=1.0, Max Tokens=512, Top-p=0.7, Top-k=50; And each term pair generates three outputs, and the average value is taken as the final score to alleviate the variability of autoregressive decoding.

8. A large language model-based medical term semantic analysis and evaluation system, characterized in that, It includes a knowledge retrieval and enrichment module, a large language model evaluation module, and a structured output generation module, each module is connected in sequence and cooperates to realize medical term semantic evaluation: The knowledge retrieval and enrichment module is used to retrieve the definitions of the medical term pairs to be evaluated and the shortest semantic path between them from the Unified Medical Language System (UMLS), and for terms with missing information in UMLS, extract supplementary definitions and external semantic paths from authoritative external sources including UpToDate and MSD manuals through the knowledge extraction tool LightRAG, and convert all semantic paths into natural language and supplement the semantic types of terms and the relationships between terms to form the basic information required for unified prompts; The large language model evaluation module includes a scorer and a supervisor, the scorer calls the large language model, generates a semantic similarity / relevance score of 1-10 and a natural language explanation of related knowledge based on the unified prompt constructed from the basic information, and the supervisor verifies the explanation and outputs or iteratively optimizes the prompt and drives the scorer to regenerate; The structured output generation module is used to integrate and output structured results including the score, natural language explanation, knowledge reference, and supervision log.

9. An electronic device, comprising: It includes: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory stores a computer program, which, when executed by the processor, causes the processor to execute the large language model-based medical term semantic analysis and evaluation method of any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed to implement the large language model-based medical term semantic analysis and evaluation method of any one of claims 1-7.

Images