A biomedical information extraction method based on a large language model
By employing a two-stage retrieval and rearrangement mechanism combining dual encoders and cross encoders, the problem of automated screening of high-quality examples in biomedical information extraction is solved, improving the accuracy and robustness of large language models and achieving efficient information extraction and knowledge graph construction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-03-05
- Publication Date
- 2026-06-09
Smart Images
Figure CN122173631A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing technology, specifically to a method for extracting biomedical information based on a large language model. Background Technology
[0002] In the field of biomedical research, automatically and accurately extracting structured knowledge (such as named entities and relations) from massive amounts of literature is crucial for building high-quality knowledge graphs and driving the development of precision medicine. In-Context Learning (ICL) has attracted widespread attention due to its ability to work without training and simply by providing task examples. However, the performance of ICL is highly dependent on the quality and relevance of the provided examples. How to automatically select and construct high-quality examples has become a core bottleneck restricting the practical application of this technology in specialized fields. Initially, high-quality examples relied entirely on manual construction by domain experts. Researchers needed to carefully design or select the most representative text fragments for specific tasks (such as "chemical disease relation extraction") and perform precise annotation. While this method can generate highly targeted examples, it has inherent drawbacks such as high cost, low efficiency, and poor scalability. It heavily relies on scarce expert knowledge, making it difficult to quickly transfer to new biomedical subfields (such as expanding from drug interactions to gene-phenotype associations), and it cannot cope with the needs of massive and diverse literature mining.
[0003] In the existing technology, in order to overcome the dependence on manual intervention, researchers have tried to optimize the way examples are presented through prompting engineering.
[0004] Yao et al. improved the extraction performance of large language models by adjusting the order of examples. (Yao Lu et al., “Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity”, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 2022, pp. 8086-8098).
[0005] Ying Mo et al. improved extraction performance by designing contrastive examples (positive and negative examples) to clarify task boundaries. (Ying Mo et al., “C-ICL: Contrastive In-context Learning for Information Extraction”, Findings of the Association for Computational Linguistics: EMNLP2024, 2024, pp.10099-10114).
[0006] Liu et al. proposed a sequence-aware example selection method, Se2 (Haoyu Liu et al., “Se2: Sequential Example Selection for In-Context Learning”, Proceedings of the 2024 Annual Conference of the Association for Computational Linguistics (ACLFindings), 2024). This method utilizes feedback from the large model itself to evaluate different contexts, thereby capturing the internal relationships and sequence information between examples and constructing more effective example sequences.
[0007] The methods described above improve the performance of ICL to some extent on a fixed set of examples, but they do not solve the fundamental problem. They are essentially only about "how to place manually selected blocks", rather than "how to automatically find more suitable blocks", and the optimization strategies themselves introduce new manual design costs, making it difficult to achieve automation and large-scale application.
[0008] On the other hand, with the popularization of the concept of Retrieval Enhanced Generation (RAG), it has become a natural approach to use automated retrieval techniques to find examples from large corpora. Existing methods are mainly based on vector retrieval models with a dual encoder architecture, which recall relevant text by calculating the similarity of semantic embeddings.
[0009] However, in the specific task of biomedical information extraction, vector retrieval models based on a dual-encoder architecture have inherent limitations: dual encoders pursue global topic similarity between texts, while the determination of biomedical entities and relationships often relies on local, fine-grained key contexts (such as specific biomarkers, drug modifiers, or causal conjunctions). This often leads to search results that are "topically related but task-irrelevant." For example, two texts discuss "lung cancer treatment," but one involves "EGFR gene mutation" and the other involves "immune checkpoint inhibitors." Although the topics are similar, they are not mutually instructive as examples for entity extraction. Furthermore, traditional retrieval cannot assess the exemplary quality and clarity of the examples themselves, and may return lengthy, vague, or distracting texts, affecting ICL performance.
[0010] In summary, current technological advancements indicate that neither manual construction nor prompting engineering has achieved automated acquisition of high-quality examples; while directly applying general retrieval techniques fails due to insufficient task relevance. Therefore, the current lack of a method to automatically and accurately filter high-quality ICL examples from the example library that possess both high semantic relevance and high task guidance constitutes a significant obstacle to the efficient and large-scale application of LLM in biomedical information extraction. Summary of the Invention
[0011] In view of this, this application discloses a biomedical information extraction method based on a large-scale language model to solve the problems of existing ICL technology, which heavily relies on manual screening of high-quality examples, resulting in high cost, low efficiency, and poor scalability. The method includes:
[0012] S1. Obtain the biomedical dataset to be processed and perform standardized preprocessing to obtain the training set; the biomedical dataset includes the named entity recognition dataset and the relation extraction dataset.
[0013] S2. Using dual encoders, the query text in the training set is converted into a high-dimensional vector containing deep semantic information, and a local vector library is built based on the high-dimensional vector.
[0014] S3. Obtain the biomedical information text to be processed as the query text; perform two-stage example retrieval and filtering on the query text in the local vector library to obtain a high-quality example set; the two-stage example retrieval and filtering includes a recall stage and a rearrangement stage.
[0015] S4. Format and arrange the high-quality example set to generate context examples; use a large language model to learn the context based on the context examples, understand the current task requirements, and generate information extraction results of the query text.
[0016] S5. Parse the information extraction results to obtain structured information extraction results.
[0017] The beneficial effects of this application include:
[0018] Based on the reordering capability of the cross-encoder model, this application designs a two-stage retrieval and reordering mechanism. It takes whether the extraction task helps the model to correctly complete the current query as the standard evaluation example, ensuring that the ICL examples provided for large language models have both high semantic relevance and high task guidance. This significantly improves the accuracy and robustness of the model in biomedical named entity recognition and relation extraction tasks, reduces the dependence on expert knowledge and manual design, and promotes the automation process of downstream applications such as knowledge graph construction.
[0019] Furthermore, dual encoder and cross encoder models (such as the BCE series) can be pre-trained or fine-tuned for biomedical texts to make them better at understanding biomedical terminology and complex sentence structures, thereby achieving more accurate example retrieval in this field.
[0020] By dynamically selecting the most relevant set of examples from the example library for different query texts, the best information extraction performance is achieved within a limited context window (e.g., using only 2-5 examples). This avoids window waste and performance degradation caused by too many or poor-quality examples, while also reducing unnecessary model computation overhead. Compared to fixed or random examples, this personalized example selection mechanism can more effectively activate the task-solving capabilities of large language models in specific contexts.
[0021] The method designed in this application can be used as an independent example preprocessing and enhancement module, and can be used in conjunction with any large language model that supports ICL, thus possessing versatility and portability. Attached Figure Description
[0022] Figure 1 This is a flowchart illustrating the biomedical information extraction method based on a large language model in this application;
[0023] Figure 2 This is a schematic diagram illustrating how the dual encoder and cross encoder calculate different types of relevant text in the embodiments of this application. Detailed Implementation
[0024] To make the objectives, technical solutions, features, and advantages of this application clearer and to enable those skilled in the art to better understand the technical solutions of this application, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments.
[0025] Example 1:
[0026] This embodiment includes a biomedical information extraction method based on a large language model, such as... Figure 1 As shown, it includes:
[0027] S1. Obtain the biomedical dataset to be processed and perform standardized preprocessing to obtain the training set; the biomedical dataset includes the named entity recognition dataset and the relation extraction dataset.
[0028] This embodiment selects publicly available biomedical datasets from the BigBio project on the HuggingFace platform. The Named Entity Recognition (NER) datasets include: BC5CDR (Chemistry and Disease Entities), NLM-Gene (Gene Entities), NCBI-Disease (Disease Entities), and Chemdner (Chemistry Entities) datasets; the Relation Extraction (RE) datasets include: BC5CDR (Chemistry-Disease Relationships), DDI Corpus (Drug-Drug Interactions), and BioRelEx (Bio-Binding Relationships) datasets.
[0029] The standardization preprocessing includes text concatenation, entity normalization, and format unification. For the j-th sample in the i-th dataset, its original text "passages" is concatenated into a unified text representation:
[0030]
[0031] in, This indicates a text concatenation operation. This indicates the number of paragraphs in the sample. The label corresponding to each paragraph in the dataset represents the correct entity extracted from that paragraph. During preprocessing, entity names need to be case-normalized and duplicates removed to obtain a standardized entity list.
[0032] Furthermore, generate structured training set files. Each sample uses the document ID as the key, and the key-value pair is a tuple containing the query text and annotations. :
[0033]
[0034] in, A unique identifier representing the k-th sample. This represents the text content of the k-th sample. The total number of samples in the dataset. This represents a standardized list of entities or relation triples.
[0035] S2. Using a dual encoder, the query text in the training set is converted into a high-dimensional vector containing deep semantic information, and a local vector library is built based on the high-dimensional vector.
[0036] The dual encoder, as a semantic encoder, is used to process text of arbitrary length. Mapped to a dense vector of fixed dimensions The process is represented as follows:
[0037]
[0038] in, This represents a dual encoder model. Indicates model parameters, Represents the vector dimension. This represents a dense vector containing text. The deep semantic information. In this embodiment, the dual encoder uses a pre-trained BCE-embedding model, and in the training dataset containing labeled results (such as entity labels and relation triples), the text portion of each sample without the results is used as a potential example.
[0039] By analyzing the entire training set Each training sample is batch encoded to obtain the vector set of the entire example library. .
[0040] Furthermore, in the construction of a local vector library based on high-dimensional vectors, to support fast nearest neighbor search, this embodiment uses the efficient vector database FAISS to analyze the set. Build an index, using cosine similarity as the index's similarity metric; for query vectors... and vectors in the library The similarity calculation formula is:
[0041]
[0042] in, This indicates similarity calculation. This represents the dot product operation. This represents the L2 norm of a vector. The closer the value is to 1, the more significant the query vector. and vectors in the library The more similar they are in the semantic space.
[0043] By transforming an unstructured text example library into an offline structured vector space that supports similarity retrieval, it can be reused once built for efficient retrieval of all new queries.
[0044] S3. Obtain the biomedical information text to be processed as the query text; perform two-stage example retrieval and filtering on the query text in the local vector library to obtain a high-quality example set; the two-stage example retrieval and filtering includes a recall stage and a rearrangement stage.
[0045] For each new query that needs to be processed (i.e., the test sample), the text content of the query text is It performs a two-stage retrieval process to select the few most instructive examples from the local vector library.
[0046] The recall phase aims to quickly filter a batch of candidate examples from the entire local example library that are relevant to the query in terms of overall topic and semantics, ensuring that no potentially relevant items are missed. Specifically: using an encoder. Encode the query text into a query vector In the local vector library FAISS index, for Perform an approximate nearest neighbor search based on a preset similarity threshold. and quantity limit Filter out those that match the query vector The search process identifies the most semantically similar candidate examples. The retrieval process can be formally defined as finding a set of vectors that satisfy the following conditions:
[0047]
[0048] Among them, set Each vector in Each corresponds to an original training sample and its text. , Select 0.79, Select 30, and As a key parameter in the retrieval process, this stage focuses on "completeness," ensuring that the candidate set covers various types of text with semantic similarity to the query.
[0049] In the reordering stage, a cross-encoder pairs the query text with each candidate example obtained in the recall stage, capturing fine-grained semantic relationships and task relevance between the text pairs. A relevance score is calculated for each <query, candidate example> pair. Based on this score, the candidate examples are sorted in descending order, and the top K examples are selected as the final high-quality ICL examples. Each example contains the original text and its corresponding standard annotation results. The reordering stage addresses the issue that the results from the recall stage may be "topically relevant but task-irrelevant." Unlike the independent encoding of dual encoders, the cross-encoder... query text and a candidate example text Simultaneously, the input is processed through a deep self-attention mechanism to determine whether the example directly helps in solving the specific information extraction task faced by the query. In this embodiment, the cross encoder is a pre-trained BCE-Reranker.
[0050] Specifically, for the candidate set Each candidate text in Construct text pairs Candidate text is quantized using a cross-encoder. mass scalar fraction The formula is:
[0051]
[0052] in, This represents the comprehensive representation vector of the [CLS] position generated by the model after interactive encoding of the input text pair; and This represents the parameters of the linear classification layer at the top of the model; This represents the Sigmoid activation function, which maps scores to the (0, 1) interval. A higher value indicates a stronger correlation.
[0053] Score for all candidate examples Perform a descending sort to obtain an ordered list:
[0054]
[0055] in, The top K examples are selected to form the final high-quality example set. In this embodiment, we select These examples not only share semantic similarity with the queries, but more importantly, they are deemed to be the most effective exemplary samples for guiding the current specific extraction task.
[0056] S4. For high-quality example sets The text is formatted and arranged to generate context examples. A large language model is used to learn the context based on these examples, understand the current task requirements, and generate information extraction results from the query text. The current task requirements include named entity recognition and relation extraction. S4 specifically includes:
[0057] S41. Based on high-quality example sets The formula for constructing prompts to guide the large language model in context learning is as follows:
[0058]
[0059] in, This indicates a prompt message. This represents a structured instruction template function, for example text. and its standard answer , Format it as a clear "instruction-output" pair.
[0060] S42. Format and concatenate the prompt information, and combine it with the query text to form a complete prompt. In this embodiment The formula is:
[0061]
[0062] in, Indicates the sequential concatenation of strings. This represents the query text.
[0063] S43, Large Language Model in Complete Prompt Below, from the previous text One example illustrates a context learning task paradigm, which generates responses to query text using an autoregressive approach. The answer is... This generation process can be understood as the model searching for the output sequence with the highest probability given prompts. The formula is:
[0064]
[0065] in, This represents the original output text generated by the large language model. This represents the conditional probability distribution defined by the language model. This represents a candidate among all possible output sequences, i.e., all possible text sequences that can be generated by traversing the vocabulary of a large language model. The large language model used in this embodiment is the pre-trained Gemma2-9B.
[0066] S5. Parse the information extraction results to obtain structured information extraction results.
[0067] Direct output of large language models It is a natural language string that needs to be parsed into structured data usable by downstream applications. For named entity recognition tasks, it identifies entity names in the information extraction results and organizes the entity names into an entity list; for relation extraction tasks, it parses the information extraction results into a list of triples of "head entity-relation-tail entity".
[0068] Furthermore, dual encoders and cross encoders are used to calculate different relevance texts, such as... Figure 2As shown in the figure, Q-P1 represents a set of low-relevance texts, and Q-P2 represents a set of high-relevance texts. The [CLS] marker is added at the beginning of each input sequence, and the output vector generated after model training aggregates the semantic information of the entire text. The [SEP] marker is used to separate different sentences. Eq(q) and Ep(p) represent the embeddings of the query and paragraph generated by the Bi-encoder architecture, which are the final processing results of the encoder. sim(q, p) represents the similarity score between the query (q) and the document (p), which is used to score and rank the matching degree between the query and the document.
[0069] For the low-relevance Q-P1 group of text, the Bi-encoder and Rerank models scored 0.842 and 0.566 respectively on the relevance of the given text group. However, for the high-relevance Q-P2 group of text, the scores were 0.636 and 0.947 respectively. This indicates that the Bi-encoder model mainly relies on character similarity for filtering and cannot effectively identify key information in the query. In contrast, the Rerank model, through the Cross-Encoder interaction mechanism, can accurately capture the key content of the query, thus significantly improving the accuracy of sample filtering.
[0070] Finally, it should be noted that the above description only depicts some embodiments of this application. For those skilled in the art, various changes, modifications, substitutions, and variations can be conceived of these embodiments without departing from the principles and spirit of this application. The scope of protection of this application is defined by the appended claims and their equivalents, and all the above-mentioned behaviors should be covered within the scope of protection of this application.
Claims
1. A biomedical information extraction method based on a large-scale language model, characterized in that, include: S1. Obtain the biomedical dataset to be processed and perform standardized preprocessing to obtain the training set; Biomedical datasets include named entity recognition datasets and relation extraction datasets; S2. Using dual encoders, the query text in the training set is converted into a high-dimensional vector containing deep semantic information, and a local vector library is built based on the high-dimensional vector. S3. Obtain the biomedical information text to be processed as the query text; perform two-stage example retrieval and filtering on the query text in the local vector library to obtain a high-quality example set; the two-stage example retrieval and filtering includes a recall stage and a rearrangement stage. S4. Format and arrange the high-quality example set to generate context examples; A large language model is used to learn context based on contextual examples, understand the current task requirements, and generate information extraction results from the query text. S5. Parse the information extraction results to obtain structured information extraction results.
2. The biomedical information extraction method based on a large language model according to claim 1, characterized in that, The dual encoder is used to map text of arbitrary length into dense vectors of fixed dimensions.
3. The biomedical information extraction method based on a large language model according to claim 2, characterized in that, The dual encoders use a pre-trained BCE-embedding model.
4. The biomedical information extraction method based on a large language model according to claim 1, characterized in that, The local vector library is constructed based on high-dimensional vectors, and cosine similarity is used as the similarity metric for the index.
5. The biomedical information extraction method based on a large language model according to claim 1, characterized in that, During the recall phase, for each new query that needs to be processed... The text content of the query text is Using an encoder query text Encoded as query vector Perform an approximate nearest neighbor search in the local vector library index, based on a preset similarity threshold. and quantity limit Filter out those that match the query vector Several candidate examples that are most similar in global semantics .
6. The biomedical information extraction method based on a large language model according to claim 5, characterized in that, During the reordering phase, the query text is processed by a cross-encoder. and For each candidate example, text pairing is performed, and a relevance score is calculated for each <query, candidate example> pair. The candidate examples are sorted in descending order based on the relevance score, and the top K examples are selected as a high-quality example set. Each example contains the original text and its corresponding standard annotation results.
7. The biomedical information extraction method based on a large language model according to claim 6, characterized in that, The cross encoder uses a pre-trained BCE-Reranker.
8. The biomedical information extraction method based on a large language model according to claim 1, characterized in that, The information extraction results of the generated query text include: S41. Based on a high-quality example set, construct prompts to guide the large language model in context learning; S42. Format and concatenate the prompt information, and combine it with the query text to form a complete prompt; S43. Given a complete prompt, the large language model uses the preceding text... One example is a context learning task paradigm that generates answers to query texts through an autoregressive approach.
9. The biomedical information extraction method based on a large language model according to claim 1, characterized in that, The information extraction results are parsed. For named entity recognition tasks, the entity names in the information extraction results are identified and organized into an entity list. For relation extraction tasks, the information extraction results are parsed into a list of triples of "head entity-relation-tail entity".