A large language model pico extraction method and system based on UMLS semantic verification

Abstract

The application belongs to the field of intelligent medical information technology, and proposes a large language model PICO extraction system and method based on UMLS semantic verification. Through multi-level cooperation of candidate medical entity generation, Transformer semantic verification, PICO label sequence legalization, UMLS semantic knowledge base semantic correction and boundary adjustment, and multi-source confidence fusion, the defects of traditional large language models in medical text PICO extraction, such as imbalance between recall rate and precision, entity boundary offset, semantic misjudgment, model illusion and poor interpretability, are effectively overcome, and high-precision and high-confidence PICO extraction of medical text is realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical information technology, and in particular to a PICO extraction system for large language models based on UMLS semantic verification. Background Technology

[0002] In evidence-based medicine and clinical research, the structured extraction of PICO elements is a crucial step in building a high-quality medical evidence system, requiring extremely high accuracy and reliability in the extracted PICO elements. However, traditional PICO identification methods often rely on sequence labeling, rule matching, or weakly supervised mechanisms. These methods generally suffer from the following problems when dealing with lengthy and structurally complex medical texts:

[0003] 1. Reliance on Annotated Data: Deep learning-based Named Entity Recognition (NER) methods rely heavily on a large amount of labeled data for training to achieve excellent recognition results. However, for rare clinical problems, insufficient PICO labeled data makes it impossible to train adequately, resulting in limited performance and an inability to extract highly accurate and reliable PICO elements from medical texts.

[0004] 2. Insufficient ability to handle cross-sentence dependencies: PICO elements often span multiple sentences, and traditional models have difficulty capturing the semantic relationships of the context, leading to incorrect labeling and consequently a decrease in the accuracy of PICO elements.

[0005] 3. Lack of semantic consistency constraints: The identification of medical terms suffers from ambiguity, misclassification, and boundary shifts, which affect the reliability of the extraction results.

[0006] 4. Poor model generalization ability: Most models rely on domain-specific corpora and have difficulty adapting to texts with different clinical backgrounds. This results in a decrease in the accuracy and credibility of the extracted PICO elements when encountering medical texts with different clinical backgrounds.

[0007] Due to the limitations mentioned above, the accuracy and reliability of PICO elements obtained by traditional PICO identification methods are difficult to guarantee, and they cannot meet the high standards required for PICO elements in evidence-based medicine and clinical research. Summary of the Invention

[0008] To address the problems in existing technologies for identifying medical PICO elements, such as reliance on labeled data, cross-sentence issues, insufficient capture of long-distance contextual dependencies, lack of semantic consistency constraints, and poor model generalization, which lead to low accuracy and reliability of extracted PICO elements, a PICO extraction method and system based on UMLS semantic verification for large language models is proposed, thus solving the aforementioned problems.

[0009] The following technical solution is adopted in this specification:

[0010] A PICO extraction method for large language models based on UMLS semantic verification, the specific steps of which are as follows:

[0011] S1: Input the medical text and structured prompts into the large language model for entity recognition, and generate a first candidate medical entity set containing candidate medical entities, BIO-PICO labels and entity confidence scores;

[0012] S2: Calculate the semantic consistency score of each candidate medical entity in the first candidate medical entity set using the Transformer validator; correct the BIO tags in the BIO-PICO tags using the structural validator, and output the second candidate medical entity set;

[0013] S3: Perform character matching and semantic matching between the second candidate medical entity set and the UMLS semantic knowledge base respectively to obtain the UMLS matching score and optimal concept of each candidate medical entity; based on the optimal concept, correct the PICO tag in the BIO-PICO tag of each candidate medical entity and adjust the boundary of each candidate medical entity to obtain the third candidate medical entity set.

[0014] S4: Perform a fusion calculation on the entity confidence score, semantic consistency score, and UMLS matching score to obtain a multidimensional fusion score; filter the candidate medical entities in the third candidate medical entity set according to the multidimensional fusion score, and output the PICO entity set.

[0015] Preferably, the structured prompts in step S1 include: the definition and examples of PICO tags, entity recognition task instructions, output format requirements, and prohibition of generating explanatory content or unstructured text.

[0016] Preferably, the method for calculating the semantic consistency score of each candidate medical entity in the first candidate medical entity using the Transformer validator in step S2 is as follows:

[0017] A1: The contextual semantic embedding vector of each candidate medical entity is obtained through a discriminant model; the discriminant model consists of an 8-layer Transformer encoder, as follows:

[0018] (1)

[0019] in, Represents the context embedding vector of the candidate medical entity. Represents the discriminant model; Indicates candidate medical entities; This represents the context sequence corresponding to the candidate medical entity in the input medical text;

[0020] A2: Input the contextual semantic embedding vector of the candidate medical entity into the discriminant head, and calculate the semantic consistency score as follows:

[0021] (2)

[0022] in, The semantic consistency score is represented by MLP, which stands for Multilayer Perceptron. This represents the activation function.

[0023] Preferably, the method for correcting the BIO tag in the BIO-PICO tag using a structure validator in step S2 is as follows:

[0024] a1: Construct an initial BIO-PICO tag sequence based on all BIO-PICO tags in the first candidate medical entity set; adjust and transform the BIO tag types in the initial BIO-PICO tag sequence to generate a BIO-PICO tag sequence set;

[0025] a2: Calculate the joint score of each BIO-PICO tag sequence in the BIO-PICO tag sequence set based on the entity confidence score output by the large language model and the BIO-PICO tag transition matrix;

[0026] a3: Sort the BIO-PICO tag sequences according to the joint score, and select the BIO-PICO tag sequence with the highest joint score, represented as:

[0027] (3)

[0028] in, This represents the BIO-PICO tag sequence with the highest joint score; Represents a set of BIO-PICO tag sequences; This represents the entity confidence score of the BIO-PICO tag corresponding to the i-th candidate medical entity in the current BIO-PICO tag sequence; This is the BIO-PICO tag transition matrix; This represents the BIO-PICO tag corresponding to the i-th candidate medical entity in the current BIO-PICO tag sequence; Indicates the first The BIO-PICO tags corresponding to each candidate medical entity in the current BIO-PICO tag sequence; This represents the total number of candidate medical entities in the first candidate medical entity set; the BIO-PICO label transition matrix stores the score for transitioning from one BIO-PICO label to another BIO-PICO label;

[0029] a4: Replace the BIO-PICO tags in the first candidate medical entity set with the BIO-PICO tag sequence with the highest joint score in order to obtain the second candidate medical entity set.

[0030] Preferably, the method for performing character and semantic matching between the second candidate medical entity set and the UMLS semantic knowledge base in step S3 is as follows:

[0031] B1: Standardize the strings of the candidate medical entities in the second candidate medical entity set to obtain the candidate medical entity strings;

[0032] B2: Perform character matching between the candidate medical entity string and the corresponding strings of each standard medical entity in the UMLS semantic knowledge base to obtain the character matching score of each standard medical entity;

[0033] B3: Perform semantic matching between the candidate medical entity strings and the strings corresponding to each standard medical entity in the UMLS semantic knowledge base to obtain the semantic matching score of each standard medical entity;

[0034] B4: The character matching score and semantic matching score are weighted and summed to obtain the medical semantic embedding fusion score for each standard medical entity.

[0035] B5: Compare the medical semantic embedding fusion scores of each standard medical entity, and take the concept of the standard medical entity with the highest medical semantic embedding fusion score as the optimal concept of the candidate medical entity; take the highest score of the medical semantic embedding fusion score as the UMLS matching score of the candidate medical entity.

[0036] Preferably, the medical semantic embedding fusion score is:

[0037] (4)

[0038] (5)

[0039] (6)

[0040] in, Indicates character matching score; This indicates that the string is being segmented; 'e' represents the candidate medical entity string; 'c' represents the string corresponding to the standard medical entity. Indicates intersection; Represents the union; Indicates semantic matching score; The semantic vector embedding representing the candidate medical entity e; The semantic vector embedding representing the standard medical entity c; Indicates cosine similarity; For fusion weights; This represents the medical semantic embedding fusion score.

[0041] Preferably, the method for correcting the PICO tags in the BIO-PICO tags of each candidate medical entity and adjusting the boundaries of each candidate medical entity based on the optimal concept in step S3 is as follows:

[0042] C1: Extract the main semantic type corresponding to the optimal concept in the UMLS semantic knowledge base;

[0043] C2: Perform PICO mapping on the main semantic type according to the preset mapping rules to obtain the PICO tag corresponding to the optimal concept; compare the PICO tag corresponding to the optimal concept with the PICO tag in the BIO-PICO tag of the candidate medical entity corresponding to the optimal concept; if they are different, replace the PICO tag in the BIO-PICO tag of the candidate medical entity with the PICO tag corresponding to the optimal concept.

[0044] C3: Calculate the string length difference between the candidate medical entity and the standard medical entity corresponding to the optimal concept; if the string length difference is greater than a preset threshold, adjust the boundary of the candidate medical entity according to the boundary of the standard medical entity to obtain the candidate medical entity with adjusted boundary; replace the candidate medical entity with the candidate medical entity with adjusted boundary.

[0045] Preferably, the calculation method for the multidimensional fusion score is as follows:

[0046] (7)

[0047] Where 'e' represents the candidate medical entity, This represents the final fusion confidence score, or multidimensional fusion score, for candidate medical entity e. This represents the sigmoid function. This represents the entity confidence level of the candidate medical entities in the third candidate medical entity set; Weighting coefficients representing entity confidence levels; The weighting coefficients representing the semantic consistency score; This represents the weighting coefficients for the UMLS matching score; Indicates the bias term; Semantic consistency scoring measures the degree of semantic matching between a candidate medical entity and its context. This indicates the UMLS matching score.

[0048] Preferably, the method for screening candidate medical entities in the third candidate medical entity set based on the multidimensional fusion score in step S4 is as follows:

[0049] When the multidimensional fusion score of a candidate medical entity is less than the preset rejection score, the candidate medical entity is a false positive and will be rejected.

[0050] When the multidimensional fusion score of a candidate medical entity is greater than or equal to the preset rejection score and less than or equal to the preset acceptance score, the candidate medical entity is a weak entity and is retained.

[0051] When the multidimensional fusion score of a candidate medical entity is greater than the preset acceptance score, the candidate medical entity is output.

[0052] A PICO extraction system for large language models based on UMLS semantic verification includes:

[0053] The semantic generator includes: a large language model that performs entity recognition on input medical text using structured prompts, wherein the large language model outputs a first set of candidate medical entities containing candidate medical entities, BIO-PICO tags, and entity confidence scores;

[0054] The validator module, connected to the semantic generator, includes: a Transformer validator, used to perform contextual semantic consistency verification on each candidate medical entity in the first candidate medical entity set and output the semantic consistency score of each candidate medical entity; and a structure validator, used to verify and correct the BIO tags in the BIO-PICO tags in the first candidate medical entity set and generate a second candidate medical entity set.

[0055] The semantic verification and normalization module, connected to the verifier module, includes: a dual matching unit, used to perform character and semantic dual matching between each candidate medical entity in the second candidate medical entity set and the UMLS semantic knowledge base to obtain the UMLS matching score and optimal concept of each candidate medical entity; and a candidate medical entity correction unit, used to correct the PICO tags in the BIO-PICO tags of each candidate medical entity in the second candidate medical entity set according to the optimal concept and adjust the entity boundaries to generate a third candidate medical entity set.

[0056] The multi-source confidence fusion module is connected to the semantic generator, the validator module, and the semantic verification and normalization module, respectively. It includes: a multi-dimensional fusion score calculation unit for obtaining the entity confidence, semantic consistency score and UMLS matching score and performing multi-dimensional fusion score calculation; and a PICO extraction result determination unit for filtering candidate medical entities in the third candidate medical entity set according to the multi-dimensional fusion score to obtain the PICO entity set.

[0057] Beneficial effects:

[0058] This invention belongs to the field of intelligent medical information technology, and proposes a PICO extraction system and method based on UMLS semantic verification of a large language model. Through multi-level collaboration including candidate medical entity generation, Transformer semantic verification, PICO tag sequence legitimization, UMLS semantic knowledge base semantic correction and boundary adjustment, and multi-source confidence fusion, it effectively overcomes the shortcomings of traditional large language models in medical text PICO extraction, such as recall-precision imbalance, entity boundary shift, semantic misjudgment, model illusion, and poor interpretability. It possesses several significant technical advantages, as detailed below:

[0059] First, this invention combines a large language model (LLM) and a UMLS semantic knowledge base to achieve high recall and high precision in PICO retrieval and extraction. Firstly, the large language model is used as a semantic generator, leveraging its strong semantic understanding capabilities to extensively mine potential PICO-related entities in the text, achieving broad coverage of candidate medical entities and ensuring high recall in entity extraction. Secondly, structured prompts are used to strictly limit the output of the large language model, avoiding irrelevant responses and improving the structured nature and interpretability of the output. Simultaneously, the UMLS medical knowledge base is introduced to accurately verify and semantically normalize the candidate medical entity set generated by the large language model, compensating for the accuracy deficiencies of the large language model in entity recognition, solving the problems of cross-sentence recognition and boundary offset in medical text PICO, and significantly improving the overall performance of medical text PICO extraction.

[0060] Second, the Transformer validator is used to measure the semantic consistency between the candidate medical entities and their corresponding contexts in the input medical text, ensuring the accuracy and reliability of the extracted candidate medical entities. The structure validator is used to perform label sequence legality verification on the candidate medical entity set, avoiding the existence of illegal BIO labels in the BIO-PICO labels in the candidate entity set, thus improving the accuracy of PICO recognition of medical text.

[0061] Third, the UMLS semantic knowledge base is introduced as a "medical judge." Then, by performing dual character and semantic matching between candidate medical entities in the second medical entity candidate set and standard medical entities in the UMLS semantic knowledge base, concept normalization, semantic type constraints, and conflict detection for high-recall medical entities are achieved. Furthermore, the candidate medical entities extracted by the large language model are adjusted using BIO-PICO tags and entity boundaries through standard medical entities in the UMLS semantic knowledge base. This not only solves the problems of concept confusion, misclassification, and medical entity offset that often occur in large language models when recognizing medical entities, but also ensures that the candidate medical entities in the entity recognition results are consistent with authoritative medical entities. Simultaneously, the introduction of the UMLS semantic knowledge base provides traceable and verifiable medical knowledge support for each PICO extraction result, enhancing the interpretability of the extraction results and improving the credibility of the model output. This addresses the problem that existing large language model PICO extraction methods often produce black-box outputs, lack effective judgment criteria, and have difficulty in tracing the credibility of the results.

[0062] Fourth, a multidimensional fusion score for candidate medical entities is calculated using entity confidence, semantic consistency score, and UMLS matching score. Based on the multidimensional fusion score and preset screening rules, candidate medical entities in the third candidate medical entity set are screened, resulting in a significant improvement in the semantic rationality and boundary accuracy of the final output PICO entity.

[0063] Fifth, this invention does not require additional training for different medical datasets or different medical subfields. It can adapt to different medical data sources and achieve high-precision and high-reliability PICO extraction of different types of medical texts simply by configuring structured prompt words and the UMLS semantic knowledge base, i.e., the medical knowledge base. Attached Figure Description

[0064] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0065] Figure 1 This is a flowchart of a PICO extraction method for a large language model based on UMLS semantic verification.

[0066] Figure 2 This is a structural diagram of a PICO extraction system for a large language model based on UMLS semantic verification. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0068] Example 1

[0069] like Figure 1 As shown, a PICO extraction method for large language models based on UMLS semantic verification includes the following steps:

[0070] S1: Input the medical text and structured prompts into the large language model for entity recognition, and generate a first candidate medical entity set containing candidate medical entities, BIO-PICO labels and entity confidence scores;

[0071] S2: Calculate the semantic consistency score of each candidate medical entity in the first candidate medical entity set using the Transformer validator; correct the BIO tags in the BIO-PICO tags using the structural validator, and output the second candidate medical entity set;

[0072] S3: Perform character matching and semantic matching between the second candidate medical entity set and the UMLS semantic knowledge base respectively to obtain the UMLS matching score and optimal concept of each candidate medical entity; based on the optimal concept, correct the PICO tag in the BIO-PICO tag of each candidate medical entity and adjust the boundary of each candidate medical entity to obtain the third candidate medical entity set.

[0073] S4: Perform a fusion calculation on the entity confidence score, semantic consistency score, and UMLS matching score to obtain a multidimensional fusion score; filter the candidate medical entities in the third candidate medical entity set according to the multidimensional fusion score, and output the PICO entity set.

[0074] like Figure 2 As shown, a PICO extraction system for large language models based on UMLS semantic verification includes:

[0075] The semantic generator includes: a large language model that performs entity recognition on input medical text using structured prompts, wherein the large language model outputs a first set of candidate medical entities containing candidate medical entities, BIO-PICO tags, and entity confidence scores;

[0076] The validator module, connected to the semantic generator, includes: a Transformer validator, used to perform contextual semantic consistency verification on each candidate medical entity in the first candidate medical entity set and output the semantic consistency score of each candidate medical entity; and a structure validator, used to verify and correct the BIO tags in the BIO-PICO tags in the first candidate medical entity set and generate a second candidate medical entity set.

[0077] The semantic verification and normalization module, connected to the verifier module, includes: a dual matching unit, used to perform character and semantic dual matching between each candidate medical entity in the second candidate medical entity set and the UMLS semantic knowledge base to obtain the UMLS matching score and optimal concept of each candidate medical entity; and a candidate medical entity correction unit, used to correct the PICO tags in the BIO-PICO tags of each candidate medical entity in the second candidate medical entity set according to the optimal concept and adjust the entity boundaries to generate a third candidate medical entity set.

[0078] The multi-source confidence fusion module is connected to the semantic generator, the validator module, and the semantic verification and normalization module, respectively. It includes: a multi-dimensional fusion score calculation unit for obtaining the entity confidence, semantic consistency score and UMLS matching score and performing multi-dimensional fusion score calculation; and a PICO extraction result determination unit for filtering candidate medical entities in the third candidate medical entity set according to the multi-dimensional fusion score to obtain the PICO entity set.

[0079] Specifically, the multidimensional fusion scoring calculation unit is connected to the semantic generator, the Transformer validator, and the dual matching unit.

[0080] Example 2

[0081] I. Core technical solution for high-recall candidate generation based on LLM:

[0082] Step 1: Use an LLM model as a semantic generator (entity recognition via LLM)

[0083] In this invention, the LLM model is considered a semantic generator according to formula (9). Its output is not the final sequence label, but rather the first candidate medical entity set. It covers potential P / I / C / O elements.

[0084]

[0085] Formula (8) is used to describe the probability distribution modeling of the LLM model generating the output sequence S under the input medical text X, providing a probabilistic basis for the subsequent generation of candidate medical entities by the large language model.

[0086]

[0087] Specifically, Structured prompts are used to compress LLM model generation through hard constraints (task definition, few-shot examples, output format requirements, prohibition of interpretable content), reduce illusions, and improve the structure and interpretability of the output.

[0088] Specifically, the PICO elements are the core information for clinical evidence retrieval and automated reviews, namely Population, Intervention, Comparison, and Outcome.

[0089] Population: The target patient population and disease characteristics of the study;

[0090] Intervention: Main intervention measures, treatment plans, etc.;

[0091] Comparison: control group, controlled intervention, placebo, etc.;

[0092] Outcome: Primary outcome measure, secondary outcome measure, etc.

[0093] Step 2: Structured prompt word design

[0094] This invention employs structured prompts to impose hard constraints on the generation of LLM models; under the constraints of structured prompts, the generation distribution of LLM models becomes... The output space is compressed to a limited number of BIO combinations. .

[0095]

[0096] Specifically, the structured prompts include: definitions and examples of the four PICO elements; explicit NER (Entity Recognition Task) instructions; fixed BIO output format requirements; and prohibition of generating explanatory content or unstructured text.

[0097] Specifically, examples of structured prompts are as follows:

[0098]

[0099] Specifically, here's an example of the results obtained after inputting structured cue words into the LLM model:

[0100] Enter medical text: A recent study evaluated metformin in type 2 diabetespatients compared to placebo. The primary outcome was HbA1c reduction.

[0101] Output:

[0102] {"entities": [{"text": "type 2 diabetes patients", "label": "BP", "confidence": 0.95},{"text": "metformin", "label": "BI", "confidence": 0.96},{"text": "placebo", "label": "BC", "confidence": 0.88},{"text": "HbA1reduction", "label": "BO", "confidence": 0.93}]}

[0103] In the output, text represents the candidate medical entity in the first candidate medical entity set; label represents the BIO-PICO label; and confidence represents the entity confidence level.

[0104] Step 3: Lightweight Transformer Validator

[0105] This invention constructs a candidate verifier (Transformer Verifier), or Transformer Verifier, using a lightweight Transformer module. The Transformer Verifier includes a discriminant model and a discriminant head.

[0106] The steps for validating the first candidate medical entity set are as follows:

[0107] The first step is to extract the context span (context text sequence) for each candidate medical entity d, using the following method:

[0108] First, based on the start and end positions of the candidate medical entity d in the original input medical text (characters or tokens), a fixed window length is extended forward and backward, centered on the candidate medical entity, to form the context text sequence containing the candidate medical entity. If the candidate medical entity is located at the text boundary or the window exceeds the range, the boundary portion is zero-padding or truncated to meet the input length requirements of the Transformer model. This context span and the context text sequence essentially refer to the same content.

[0109] The second step involves inputting the candidate medical entity and its corresponding context text sequence into the candidate validator, either the Verifier or the Transformer, to obtain a continuous discrimination score. They decide whether to retain the candidate.

[0110] Specifically, the steps in the Transformer validator are as follows:

[0111] 1) Input the candidate medical entity and the context text sequence containing the candidate medical entity into the discriminative model to generate a context semantic embedding vector, as follows:

[0112] (1)

[0113] in, Represents the context embedding vector of the candidate medical entity. The discriminant model consists of 8 layers of transformer encoders; The context text sequence in which the candidate medical entity is located. Candidate medical entities;

[0114] 2) Input the semantic embedding into the discriminant head to obtain the semantic consistency discrimination score, which is:

[0115] (2)

[0116] in, The semantic consistency score is represented by MLP, which stands for Multilayer Perceptron. This represents the activation function.

[0117] Step 4: Sequence Structure Legalization

[0118] The correct labeling of BIO-PICO tags directly corresponds to the logical chain of medical problems and the core reasoning rules of evidence-based medicine. Incorrect labeling can disrupt the inherent relationships between PICO elements, leading to deviations throughout the entire process of information extraction, evidence retrieval, and clinical decision-making. Therefore, it is necessary to verify the legitimacy of the BIO-PICO tag paths in the first candidate medical entity set extracted by the large language model.

[0119] The Verifier verification in Step 3 targets the semantic consistency of candidate medical entities and does not involve the verification of the legality of PICO tag paths. Moreover, the BIO tags in the BIO-PICO tags generated by LLM are not entirely correct and often contain structurally invalid states, such as BP→O→IP. To ensure the legality of BIO tag paths in BIO-PICO tags, a CRF-style structural constraint, i.e., a CRF model, is introduced to correct the BIO tags in BIO-PICO tags. Structural legality constraints are applied according to formula (3), i.e., it is determined whether the tag transfer between medical entities in the candidate medical entity set conforms to the BIO rules.

[0120] Specifically, the steps for global validation and correction of the BIO tag sequence using CRF-style structural constraints are as follows:

[0121] Extract BIO tags from all BIO-PICO tags in the first candidate medical entity set to construct an initial BIO-PICO tag sequence; adjust and modify the BIO tag types in the initial BIO-PICO tag sequence to generate a BIO-PICO tag sequence set;

[0122] Specifically, the adjustment involves performing a full enumeration transformation on the BIO tags in the BIO-PICO tags, changing them from "B to I" or "I to B".

[0123] The entity confidence scores output by the large language model are added to the transition scores in the transition matrix to obtain a joint score for each label sequence, which comprehensively reflects the overall rationality of the current BIO label sequence. All label sequences are then sorted based on the joint score, and the label sequence with the highest score is selected as the final label sequence.

[0124] (3)

[0125] in, This represents the BIO-PICO tag sequence with the highest joint score; Represents a set of BIO-PICO tag sequences; This represents the entity confidence score of the BIO-PICO tag corresponding to the i-th candidate medical entity in the current BIO-PICO tag sequence; This is the BIO-PICO tag transition matrix; This represents the BIO-PICO tag corresponding to the i-th candidate medical entity in the current BIO-PICO tag sequence; Indicates the first The BIO-PICO tags corresponding to each candidate medical entity in the current BIO-PICO tag sequence; This represents the total number of candidate medical entities in the first candidate medical entity set; the BIO-PICO label transfer matrix stores the score for transferring from one BIO-PICO label to another BIO-PICO label.

[0126] Specifically, the label transfer matrix is ​​manually set to determine which label transfers are reasonable and which are unreasonable.

[0127] Specifically, CRF-style structural constraints add a "security check" to the label sequence of candidate medical entity sets to ensure the legality and compliance of the generated content structure. Its core is a "label transition matrix" that dynamically constrains the data: the matrix assigns high scores to legitimate label transitions and low scores to illegitimate ones, thus restricting which labels can be linked consecutively and which absolutely cannot overlap. CRF-style structural constraints enforce entity integrity (e.g., BI must be followed by II to avoid unclosed intervention label closures), ultimately ensuring the compliance of the sequence.

[0128] Step 5: Output of Candidate Medical Entities

[0129] The output of this stage is the second candidate medical entity set. This set has the advantages of broad semantic coverage and strong entity capture capabilities. However, it still suffers from three typical defects: entity boundary offset, semantic type ambiguity, and domain / general confusion. Therefore, standardization and posterior semantic correction are necessary for the next phase of the UMLS medical knowledge base.

[0130] II. Core Technical Solutions for Semantic Mapping and Entity Normalization Post-processing Semantic Correction Based on the Medical Domain Knowledge Base UMLS

[0131] Using the UMLS semantic knowledge base as a semantic filter, candidate medical entities are mapped to standard medical concepts. Semantic correction is performed using the standard structure of UMLS, that is, the PICO tags in the BIO-PICO tags are corrected.

[0132] The UMLS semantic knowledge base includes a super thesaurus, a semantic network, and an expert dictionary; among them, the super thesaurus is responsible for integrating multi-source medical terms, and each medical data has a corresponding concept definition and semantic classification.

[0133] Step 1: Dual matching mechanism of characters and semantics

[0134] Candidate medical entities are standardized as strings and then compared with the UMLS semantic knowledge base. Perform double matching. The matching method is obtained by matching the intersection of lexical-level normalized sets according to formula (4). Semantic vector matching is performed to obtain Finally, using formula (6) for medical semantic embedding fusion scoring, the optimal concept is selected:

[0135] (4)

[0136] (5)

[0137] (6)

[0138] The 1 in the semantic vector matching formula is used for smoothing, making the fusion range score range more stable (avoiding score abnormalities when the cosine similarity is negative).

[0139] Traverse all standard medical entities in the UMLS semantic knowledge base, and use the concept of the standard medical entity with the highest medical semantic embedding and fusion score as the optimal concept of the candidate medical entity:

[0140] (11)

[0141] in, U represents the optimal concept, and U represents the set of all standard medical entity strings in the UMLS semantic knowledge base.

[0142] Step 2: Mapping the concept of semantic type from UMLS to PICO

[0143] Successfully matched the optimal concept in UMLS Then, a deterministic mapping is performed based on its primary semantic type, so the mapping function is formally set as Equation (12).

[0144]

[0145] in, This represents the mapping function, which specifies the mapping rules from the optimal concept to the PICO label. It represents the concept of optimality.

[0146] Specifically, the deterministic mapping relationship This is based on the clinically relevant mapping rules between the Primary Semantic Type (PST) and the four PICO elements, officially released by UMLS. These rules are pre-defined and solidified by medical experts using the UMLS Semantic Network. An example is shown below: ; ;

[0147] If the same entity matches multiple semantic types, the highest value is taken according to the clinical priority, where the clinical priority is:

[0148]

[0149] Clinical priority means that when a candidate medical entity can be mapped to multiple PICO categories simultaneously, the selection is based on the following order: intervention (I) has the highest priority, followed by control (C), then population (P), and finally outcome (O). For example, if an entity matches both "Drug" (belonging to I) and "Disease" (belonging to P), it will ultimately be mapped to category I.

[0150] Step 3: Boundary Optimization

[0151] Candidate medical entities are standardized using a UMLS terminology vocabulary before undergoing precise and partial matching. For entities that match successfully but have abnormal boundaries, boundary fine-tuning is performed based on the vocabulary length and context alignment rules of the UMLS standard form, correcting common truncation or over-inclusion issues in large models. The specific method is as follows:

[0152] Determine if the boundary is abnormal: Compare the string length len(e) of the candidate medical entity e with the length len(c) of the standard medical entity c corresponding to the best UMLS concept it matches;

[0153] like If the boundary of a candidate medical entity contains obvious non-medical words / punctuation / connecting words, or breaks a complete medical phrase, it is judged as a boundary abnormality.

[0154] Boundary fine-tuning rules: If the standard medical entity c corresponding to the optimal UMLS concept is longer than the string length of the candidate medical entity e, then the entity boundary is expanded to the left / right until the lexical boundary of the candidate medical entity e is aligned with that of c or the semantic consistency of the context is highest; if e is too long, then it is shrunk according to the lexical boundary of c, and the semantic consistency score of the shrunk segment is verified; after fine-tuning, the UMLS matching score is recalculated, and if the score drops significantly, it is reverted to the original boundary.

[0155] Step 3: Multi-source confidence fusion

[0156] The final entity is multi-source signal fusion: LLM generates confidence scores. Verifier semantic consistency UMLS matching strength The final result can be derived from formula (7).

[0157] (7)

[0158] Where 'e' represents the candidate medical entity, This represents the final fusion confidence score, or multidimensional fusion score, for candidate medical entity e. This represents the sigmoid function. This represents the entity confidence level of the candidate medical entities in the third candidate medical entity set; Weighting coefficients representing entity confidence levels; The weighting coefficients representing the semantic consistency score; This represents the weighting coefficients for the UMLS matching score; Indicates the bias term; Semantic consistency scoring measures the degree of semantic matching between a candidate medical entity and its context. This indicates the UMLS matching score.

[0159] High-recall entities in the high-recall entity set are filtered according to a preset filtering rule. The preset filtering rule is as follows:

[0160] False positives were eliminated. If identified as a weak entity, it can be selectively retained based on actual application needs, and the threshold can be dynamically adjusted according to downstream task requirements (such as automated literature reviews or human-assisted review). and This enhances the system's practicality and adaptability; : Output the final entity.

[0161] Specifically, This indicates the preset elimination score; This indicates that the default rating is accepted.

[0162] Candidate medical entities that fail to match UMLS concepts are considered potential false positives and are either eliminated or relabeled based on the contextual confidence threshold. After the above processing, the final output PICO entities show significant improvements in semantic reasonableness and boundary accuracy, thereby improving the overall Precision (accuracy score) and F1-score (F1 score) of the output results.

[0163] Example 3

[0164] This embodiment uses the Cochrane dataset as an example to deploy the patented method for PICO extraction experiments, demonstrating the application effect of the device in real medical texts. The score values ​​listed in this embodiment are exemplary implementation data used to illustrate the execution flow and judgment logic of the method of the present invention.

[0165] 1. Enter medical text:

[0166] “A recent double-blind, randomized, placebo-controlled study evaluated the efficacy of metformin in type 2 diabetes patients compared to a standard diet regime over 12 months. The primary outcome was HbA1c reduction, with secondary endpoints including weight loss and adverse events.”

[0167] 2. LLM Candidate Generation

[0168] This paper utilizes the Qwen3-32B large language model to generate candidate medical entities, providing structured prompts (including PICO definitions, examples, and output format constraints) to generate these candidate medical entities.

[0169] • Population: “type 2 diabetes patients”

[0170] • Intervention: “metformin”

[0171] • Comparison: “placebo-controlled”

[0172] • Outcome: “efficacy”, “HbA1c reduction”

[0173] The model generates confidence scores for each candidate medical entity and its category, as shown in Table 1.

[0174] Table 1: Candidate Medical Entities Generated by LLM and Their Confidence Levels

[0175]

[0176] Among them, entities such as "metformin" and "type 2 diabetes patients" have high stability in the generation process, while "efficacy" has relatively low generation confidence due to semantic abstraction.

[0177] 3. Transformer validator processing

[0178] For each candidate medical entity, the context span is extracted, and a lightweight 8-layer Transformer encoder is used to generate semantic embeddings. Semantic consistency scores are then calculated, and context embeddings are generated and their credibility is calculated. At this stage, the semantic consistency scores for each candidate medical entity are shown in Table 2.

[0179] Table 2: Semantic Consistency Scoring of Transformer Validator

[0180]

[0181] Among them, "efficacy" has a more general meaning, but it is closely related to the description of the outcome in the context, so it still received a high consistency score.

[0182] 4. UMLS verification and normalization

[0183] After string standardization (case normalization and word form restoration), candidate medical entities are subjected to dual matching with UMLS Metathesaurus: character matching and semantic vector matching. The optimal concept is selected by fusing the scores. For candidate medical entities that fail to map to a unique stable CUI, their UMLS matching score can be calculated from methodological semantics, similarity to related concepts, or auxiliary knowledge features.

[0184] • “metformin” matches the UMLS terminology database and is normalized to the standard term C0026212 (Metformin Hydrochloride), with a semantic type of Drug → I (Intervention).

[0185] • “standard diet regime” matches Procedure → C (Comparison);

[0186] • “Type 2 diabetes” is mapped to Disease / Syndrome → P (Population);

[0187] • “HbA1c reduction” matches as Laboratory / Test Result → O (Outcome);

[0188] “efficacy” failed to match the main semantic type of UMLS exactly, but due to the strong semantic context, it was reserved as O (Outcome).

[0189] Perform UMLS concept mapping on candidate medical entities and calculate the knowledge matching score between candidate medical entities and the optimal medical concept. The UMLS validation results and scores for each candidate medical entity are shown in Table 3:

[0190] Table 3: UMLS Semantic Validation and Normalized Scoring

[0191]

[0192] Among them, "placebo-controlled" essentially describes the experimental design, and there is often no stable single CUI in UMLS; "efficacy" cannot be accurately mapped to a single UMLS main semantic type, and its knowledge matching score is significantly lower than other entities, but it still enters the fusion stage, and its final fate is determined by the scores from multiple sources.

[0193] The CUI number shown in this embodiment is used to illustrate the medical concept normalization process of candidate medical entities. The specific concept identifier may change with the UMLS version.

[0194] 5. Multi-source confidence fusion

[0195] The confidence scores of each module are weighted and fused; the weighted fusion formula is as follows:

[0196] (7)

[0197] The threshold is set as follows: = 0.3, = 0.7, finally determining that the entity is retained and generating a PICO tag.

[0198] In this embodiment, the fused scoring result is as follows:

[0199] Table 4 Multi-source confidence fusion score

[0200]

[0201] Although "efficacy" scored low in the UMLS verification phase, it scored high in the generation and semantic consistency phases, so its final credibility still exceeded the acceptance threshold and was thus retained as Outcome.

[0202] 6. Output Results

[0203] Finally, the device outputs the PICO extraction results in JSON format, along with semantic context and a credibility score:

[0204]

[0205] Example 4

[0206] The processing flow of a PICO extraction system for large language models based on UMLS semantic verification is as follows:

[0207] 1. Data Input

[0208] The input source is unstructured clinical text, including literature abstracts, systematic review texts, or clinical trial reports.

[0209] 2. Candidate Entity Generation Module

[0210] 1) The text undergoes paragraph filtering and sentence segmentation, standardization cleaning, and removal of irrelevant symbols and punctuation.

[0211] 2) Utilize LLM as a semantic generator to provide structured cue words to LLM;

[0212] 3) Obtain the BIO format candidate entity set and its confidence score;

[0213] 3. Verifier Module

[0214] Semantic embedding: for each candidate medical entity The context span (context text sequence) is extracted as follows: First, based on the start and end positions of the characters or tokens of the candidate medical entity d in the original input medical text, a fixed window length is extended forward and backward, centered on the entity, thus forming the context text sequence X of the entity. If the entity is located at the text boundary or the window exceeds the range, zero-padding or truncation is applied to the boundary portion to meet the input length requirements of the Transformer model. This context span and the context text sequence essentially refer to the same content.

[0215] Validators: including Transformer validators and structural validators.

[0216] The Transformer validator is used for discriminative score verification: it uses a lightweight 8-layer Transformer encoder to perform semantic embedding on candidate entities and their context; it is used to calculate the semantic consistency between the entity and the context, and outputs a semantic credibility score, i.e., a semantic consistency score.

[0217] The structure validator is used for sequence structure legality verification: the validator filters out the BIO sequences with illegal structures generated by LLM and obtains a high-recall candidate entity set, namely the second candidate medical entity set. Each candidate medical entity in the second candidate medical entity set has a corresponding semantic credibility score, namely the semantic consistency score.

[0218] 4. UMLS semantic verification and normalization module

[0219] 1) Standardize the candidate medical entities in the high-recall candidate entity set to obtain candidate medical entity strings;

[0220] 2) Perform dual character and semantic matching between the candidate medical entity strings and the UMLS terminology database to obtain the UMLS semantic strength score of the candidate medical entity, i.e., the medical semantic embedding fusion score; determine the optimal concept of the candidate medical entity based on the medical semantic embedding fusion score;

[0221] 3) Determine the PICO tag of the optimal concept based on the main semantic type corresponding to the optimal concept in the UMLS semantic knowledge base, based on the main semantic type of the optimal concept and the preset mapping function;

[0222] 5. Multi-source confidence fusion module

[0223] 1) The confidence scores of LLM-generated entities, Transformer validator confidence scores, and UMLS semantic matching strength scores are weighted and fused to generate the final confidence score;

[0224] 2) Filter all candidate medical entities based on the final confidence level: remove false positives, retain weak entities, and output high-confidence entities;

[0225] 3) The candidate medical entities are standardized using the terminology lexicon in UMLS, followed by exact and partial matching. For entities that match successfully but have abnormal boundaries, the boundaries are fine-tuned according to the vocabulary length and context alignment rules of the UMLS standard surface form, correcting common truncation or over-inclusion problems in large models.

[0226] Boundary anomaly optimization for candidate medical entities can be performed after threshold filtering, or after semantic type mapping in the semantic verification and normalization module, and before calculating the multi-source confidence fusion score. However, whether the boundary adjustment of candidate medical entities is performed before or after calculating the multi-source confidence fusion score, it does not affect the accuracy and reliability of the final PICO extraction results.

[0227] Output the final extracted PICO four-element entities, providing entity type labels and text context information.

[0228] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for extracting PICO from a large language model based on UMLS semantic verification, characterized in that, S1: Input the medical text and structured prompts into the large language model for entity recognition, and generate a first candidate medical entity set containing candidate medical entities, BIO-PICO labels and entity confidence scores; S2: Calculate the semantic consistency score of each candidate medical entity in the first candidate medical entity set using the Transformer validator; correct the BIO tags in the BIO-PICO tags using the structural validator, and output the second candidate medical entity set; S3: Perform character matching and semantic matching between the second candidate medical entity set and the UMLS semantic knowledge base respectively to obtain the UMLS matching score and optimal concept of each candidate medical entity; based on the optimal concept, correct the PICO tag in the BIO-PICO tag of each candidate medical entity and adjust the boundary of each candidate medical entity to obtain the third candidate medical entity set. S4: The entity confidence score, semantic consistency score and UMLS matching score are fused and calculated to obtain a multi-dimensional fusion score; Based on the multidimensional fusion score, candidate medical entities in the third candidate medical entity set are screened, and a PICO entity set is output.

2. The PICO extraction method for large language models based on UMLS semantic verification as described in claim 1, characterized in that, The structured prompts mentioned in step S1 include: the definition and examples of PICO tags, entity recognition task instructions, output format requirements, and prohibition of generating explanatory content or unstructured text.

3. A method for extracting PICO from a large language model based on UMLS semantic verification as described in claim 1, characterized in that, The method for calculating the semantic consistency score of each candidate medical entity in the first candidate medical entity using the Transformer validator in step S2 is as follows: A1: The contextual semantic embedding vector of each candidate medical entity is obtained through a discriminant model; the discriminant model consists of an 8-layer Transformer encoder, as follows: （1） in, Represents the context embedding vector of the candidate medical entity. Represents the discriminant model; Indicates candidate medical entities; This represents the context sequence corresponding to the candidate medical entity in the input medical text; A2: Input the contextual semantic embedding vector of the candidate medical entity into the discriminant head, and calculate the semantic consistency score as follows: （2） in, The semantic consistency score is represented by MLP, which stands for Multilayer Perceptron. This represents the activation function.

4. A method for extracting PICO from a large language model based on UMLS semantic verification as described in claim 1, characterized in that, The method for correcting the BIO tag in the BIO-PICO tag using the structure validator described in step S2 is as follows: a1: Construct an initial BIO-PICO tag sequence based on all BIO-PICO tags in the first candidate medical entity set; adjust and transform the BIO tag types in the initial BIO-PICO tag sequence to generate a BIO-PICO tag sequence set; a2: Calculate the joint score of each BIO-PICO tag sequence in the BIO-PICO tag sequence set based on the entity confidence score output by the large language model and the BIO-PICO tag transition matrix; a3: Sort the BIO-PICO tag sequences according to the joint score, and select the BIO-PICO tag sequence with the highest joint score, represented as: （3） in, This represents the BIO-PICO tag sequence with the highest joint score; Represents a set of BIO-PICO tag sequences; This represents the entity confidence score of the BIO-PICO tag corresponding to the i-th candidate medical entity in the current BIO-PICO tag sequence; This is the BIO-PICO tag transition matrix; This represents the BIO-PICO tag corresponding to the i-th candidate medical entity in the current BIO-PICO tag sequence; Indicates the first The BIO-PICO tags corresponding to each candidate medical entity in the current BIO-PICO tag sequence; This represents the total number of candidate medical entities in the first candidate medical entity set; the BIO-PICO label transition matrix stores the score for transitioning from one BIO-PICO label to another BIO-PICO label; a4: Replace the BIO-PICO tags in the first candidate medical entity set with the BIO-PICO tag sequence with the highest joint score in order to obtain the second candidate medical entity set.

5. A method for extracting PICO from a large language model based on UMLS semantic verification as described in claim 1, characterized in that, The method for performing character and semantic matching between the second candidate medical entity set and the UMLS semantic knowledge base in step S3 is as follows: B1: Standardize the strings of the candidate medical entities in the second candidate medical entity set to obtain the candidate medical entity strings; B2: Perform character matching between the candidate medical entity string and the corresponding strings of each standard medical entity in the UMLS semantic knowledge base to obtain the character matching score of each standard medical entity; B3: Perform semantic matching between the candidate medical entity strings and the strings corresponding to each standard medical entity in the UMLS semantic knowledge base to obtain the semantic matching score of each standard medical entity; B4: The character matching score and semantic matching score are weighted and summed to obtain the medical semantic embedding fusion score for each standard medical entity. B5: Compare the medical semantic embedding fusion scores of each standard medical entity, and take the concept of the standard medical entity with the highest medical semantic embedding fusion score as the optimal concept of the candidate medical entity; take the highest score of the medical semantic embedding fusion score as the UMLS matching score of the candidate medical entity.

6. A method for extracting PICO from a large language model based on UMLS semantic verification as described in claim 5, characterized in that, The medical semantic embedding fusion score is as follows: （4） （5） （6） in, Indicates character matching score; This indicates that the string is being segmented; 'e' represents the candidate medical entity string; 'c' represents the string corresponding to the standard medical entity. Indicates intersection; Represents the union; Indicates semantic matching score; The semantic vector embedding representing the candidate medical entity e; The semantic vector embedding representing the standard medical entity c; Indicates cosine similarity; For fusion weights; This represents the medical semantic embedding fusion score.

7. A method for extracting PICO from a large language model based on UMLS semantic verification as described in claim 1, characterized in that, The method for correcting the PICO tags in the BIO-PICO tags of each candidate medical entity and adjusting the boundaries of each candidate medical entity based on the optimal concept in step S3 is as follows: C1: Extract the main semantic type corresponding to the optimal concept in the UMLS semantic knowledge base; C2: Perform PICO mapping on the main semantic type according to the preset mapping rules to obtain the PICO tag corresponding to the optimal concept; compare the PICO tag corresponding to the optimal concept with the PICO tag in the BIO-PICO tag of the candidate medical entity corresponding to the optimal concept; if they are different, replace the PICO tag in the BIO-PICO tag of the candidate medical entity with the PICO tag corresponding to the optimal concept. C3: Calculate the string length difference between the candidate medical entity and the standard medical entity corresponding to the optimal concept; If the string length difference is greater than a preset threshold, the boundary of the candidate medical entity is adjusted according to the boundary of the standard medical entity to obtain a candidate medical entity with adjusted boundary; the candidate medical entity is then replaced with the candidate medical entity with adjusted boundary.

8. A method for extracting PICO from a large language model based on UMLS semantic verification as described in claim 1, characterized in that, The calculation method for the multidimensional fusion score is as follows: （7） Where 'e' represents the candidate medical entity, This represents the final fusion confidence score, or multidimensional fusion score, for candidate medical entity e. This represents the sigmoid function. This represents the entity confidence level of the candidate medical entities in the third candidate medical entity set; Weighting coefficients representing entity confidence levels; The weighting coefficients representing the semantic consistency score; This represents the weighting coefficients for the UMLS matching score; Indicates the bias term; Score semantic consistency. This indicates the UMLS matching score.

9. A method for extracting PICO from a large language model based on UMLS semantic verification as described in claim 1, characterized in that, The method for screening candidate medical entities in the third candidate medical entity set based on the multidimensional fusion score in step S4 is as follows: When the multidimensional fusion score of a candidate medical entity is less than the preset rejection score, the candidate medical entity is a false positive and will be rejected. When the multidimensional fusion score of a candidate medical entity is greater than or equal to the preset rejection score and less than or equal to the preset acceptance score, the candidate medical entity is a weak entity and is retained. When the multidimensional fusion score of a candidate medical entity is greater than the preset acceptance score, the candidate medical entity is output.

10. A PICO extraction system for large language models based on UMLS semantic verification, formed by the method of any one of claims 1-9, characterized in that, The semantic generator includes: a large language model that performs entity recognition on input medical text using structured prompts, wherein the large language model outputs a first set of candidate medical entities containing candidate medical entities, BIO-PICO tags, and entity confidence scores; The validator module, connected to the semantic generator, includes: a Transformer validator, used to perform contextual semantic consistency verification on each candidate medical entity in the first candidate medical entity set and output the semantic consistency score of each candidate medical entity; and a structure validator, used to verify and correct the BIO tags in the BIO-PICO tags in the first candidate medical entity set and generate a second candidate medical entity set. The semantic verification and normalization module, connected to the verifier module, includes: a dual matching unit, used to perform character and semantic dual matching between each candidate medical entity in the second candidate medical entity set and the UMLS semantic knowledge base to obtain the UMLS matching score and optimal concept of each candidate medical entity; and a candidate medical entity correction unit, used to correct the PICO tags in the BIO-PICO tags of each candidate medical entity in the second candidate medical entity set according to the optimal concept and adjust the entity boundaries to generate a third candidate medical entity set. The multi-source confidence fusion module is connected to the semantic generator, the validator module, and the semantic verification and normalization module, respectively. It includes: a multi-dimensional fusion score calculation unit for obtaining the entity confidence, semantic consistency score and UMLS matching score and performing multi-dimensional fusion score calculation; and a PICO extraction result determination unit for filtering candidate medical entities in the third candidate medical entity set according to the multi-dimensional fusion score to obtain the PICO entity set.

Images