Biomedicine named entity recognition method based on multi-corpus aggregation and large model corpus expansion

Abstract

The invention discloses a biomedicine named entity recognition method based on multi-corpus aggregation and large model corpus expansion. The method mainly comprises the steps of multi-corpus aggregation based on BERT-BiLSTM-CRF and corpus expansion based on a large language model. According to the BERT-BiLSTM-CRF-based multi-corpus aggregation, firstly, text data of a plurality of biomedical corpora are collected and screened, corpus preprocessing and aggregation operation are carried out through a BIO annotation format, and an aggregation strategy comprises direct aggregation and an aggregation mode for marking corpus sources. Corpus expansion based on a large language model aggregates corpora through a cue word mechanism, and performs model training in combination with a LoRA technology to generate expanded corpus data. And the expanded corpus is re-processed and then re-aggregated with the original corpus, and finally input into the BERT-BiLSTM-CRF model for named entity recognition. Through the combination of multi-corpus aggregation and corpus expansion, the method can significantly improve the efficiency and accuracy of named entity recognition, and enhance the generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and biomedical information processing, and in particular to a biomedical named entity recognition method based on multi-corpus aggregation and large model expansion. Background Art

[0002] With the rapid development of biomedical information, massive amounts of text data have brought challenges to the extraction of named entities such as diseases, drugs, and symptoms. Traditional rule-based and dictionary-based named entity recognition methods can no longer meet the needs of modern information extraction due to their lack of ability to adapt to complex biomedical texts. In recent years, deep learning techniques (such as BERT, BiLSTM, and CRF) have been widely used in named entity recognition tasks and have achieved good results. However, such methods usually rely on a single corpus for training, and the model's generalization ability is limited, making it difficult to process different data from multiple corpora.

[0003] At the same time, large-scale pre-trained language models (such as Qwen2-7b, Llama3-8b, Gemma2-9b) have demonstrated strong context understanding and corpus generation capabilities, especially when combined with the prompt word mechanism, they can effectively cope with named entity recognition tasks in a multi-corpus environment. How to combine traditional deep learning models with large model corpus expansion technology to improve the generalization ability of the model in a multi-corpus environment has become the current research focus. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to propose a biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion, which generates high-quality expanded corpus through a large model, further aggregates it with the original corpus, and combines it with the BERT-BiLSTM-CRF model for final entity recognition, thereby improving the accuracy and robustness of the recognition task.

[0005] Technical solution: To achieve the above-mentioned invention object, the present invention provides a biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion, comprising the following steps:

[0006] Collect text data from multiple public biomedical corpora, perform text cleaning and annotate named entities in the text according to BIO format;

[0007] Add source tags before and after each sentence to clearly indicate the corpus from which the sentence comes;

[0008] The corpus is expanded based on the big model through the prompt word mechanism. The prompt words clearly indicate the task description and the output JSON format. At the same time, the source of each corpus is clearly marked in the prompt words, which helps the big model understand the characteristics of different corpora and generate expanded corpora with different corpus styles.

[0009] After the extended corpus with BIO annotation format generated by the large model according to the prompt words is aggregated with the original corpus, it is input into the BERT-BiLSTM-CRF model for final training, and biomedical named entity recognition is performed through the CRF layer. Through the joint training of the cross entropy loss function and the conditional random field, the log-likelihood function of the entity label sequence is maximized, thereby optimizing the biomedical named entity recognition performance.

[0010] Furthermore, the prompt words for expanding the corpus through the large model are designed as follows: first, clarify the task goal, tell the model the source of the sentence, and require the model to generate new named entity recognition data, indicating the entity category in the task; then, provide specific requirements for input and output, and ensure that the output strictly follows the following JSON format: {"text":"sentence","outputs":"entities in the sentence marked with <type>tags< / type> "}; At the same time, it is required to indicate the source of the text; To avoid generating duplicate or similar content, it is clearly stated that the examples are only used to illustrate the output format; Finally, examples are used to show how to use labels to annotate entity categories in the output.

[0011] Furthermore, before generating the extended corpus, the large model is fine-tuned and trained using the LoRA technology, so that the model can better adapt to the original corpus generation process guided by the prompt words.

[0012] Furthermore, in the process of fine-tuning the large model of LoRA, the loss function adopts cross entropy loss, and the loss function is:

[0013]

[0014] Where L represents the cross entropy loss, which is used to calculate the deviation of the model from the true label when predicting the entity type of each word or phrase. N represents the total number of samples involved in the calculation of the cross entropy loss. i is the true label, corresponding to the correct entity type annotation for each word or phrase in the text, The label probability predicted by the model indicates the model's prediction result for the entity type of each word or phrase, and gives the model's confidence in each possible category in the form of probability.

[0015] Furthermore, in the BERT-BiLSTM-CRF model, the BERT model is used as the basis for text feature extraction. BERT learns the contextual dependencies in the text through a bidirectional Transformer structure and generates deep semantic feature representations. Based on the feature representation generated by BERT, a bidirectional long short-term memory network BiLSTM is used to further process the sequence, and semantic representations in two directions are generated through forward and reverse LSTM units. Conditional random fields CRF are used to annotate the final features.

[0016] Furthermore, when training the BERT-BiLSTM-CRF model, a multi-corpus joint loss function is defined to perform weighted optimization on the data of different corpora. The joint loss function is:

[0017]

[0018] Where N is the number of corpora, α i is the weight of the i-th corpus, L i is the loss function of the i-th corpus, P i (y|x) is the predicted probability of the i-th corpus, y i is the true label of the i-th corpus.

[0019] Furthermore, in the process of fine-tuning the large model and generating expanded corpus, the prompt words are dynamically adjusted and experimented. Through prompt word ablation experiments, the impact of prompt words on model performance is evaluated to find the optimal prompt word design.

[0020] Furthermore, after the BERT-BiLSTM-CRF model is trained, the performance of the model is evaluated on an independent validation set. The entity recognition ability of the model is measured by the F1 score, precision, and recall. Based on the evaluation results of the model on the validation set, iterative optimization is performed, including:

[0021] Adjust the model structure: adjust the number of layers and hidden units of BERT, and the number of layers and hidden dimensions of BiLSTM to improve the expressiveness of the model;

[0022] Optimize hyperparameters: Use grid search or Bayesian optimization methods to adjust learning rate, batch size, weight decay, and find the optimal configuration of the model;

[0023] Regularization strategy: Introduce Dropout and L2 regularization methods to prevent model overfitting and improve the generalization ability of the model;

[0024] Early stopping strategy: When the performance of the validation set no longer improves, the training is terminated early to avoid the model overfitting the training data;

[0025] Data enhancement: Use data enhancement technology to increase the diversity of training data and improve the model's adaptability to unknown data.

[0026] Based on the same inventive concept, the present invention provides a biomedical named entity recognition system based on multi-corpus aggregation and large model corpus expansion, comprising:

[0027] The multi-corpus aggregation module based on BERT-BiLSTM-CRF is used to collect text data from multiple public biomedical corpora, perform text cleaning and annotate named entities in the text according to the BIO format; add corpus source labels before and after each sentence to clearly indicate the corpus from which the sentence comes; and aggregate the expanded corpus generated by the corpus expansion module based on the large language model with the original corpus, input it into the BERT-BiLSTM-CRF model for final training, and perform biomedical named entity recognition through the CRF layer. Through the joint training of the cross entropy loss function and the conditional random field, the log-likelihood function of the entity label sequence is maximized to optimize the biomedical named entity recognition performance;

[0028] And the corpus expansion module based on large language models is used to expand the corpus based on the large model through the prompt word mechanism. The prompt words clearly explain the task and output the JSON format. At the same time, the source of each corpus is clearly marked in the prompt words, which helps the large model understand the characteristics of different corpora and generate expanded corpora with different corpus styles.

[0029] The present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion.

[0030] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0031] (1) The present invention can improve the generalization ability of the model by jointly training BERT-BiLSTM-CRF with multiple corpora, and show stronger adaptability and robustness when processing data from multiple corpora.

[0032] (2) The present invention optimizes the fine-tuning process of the large model through the LoRA technology, greatly reducing the number of parameters for model training, enhancing the adaptability of the large model to different corpora, and being able to generate high-quality expanded corpora consistent with the original corpus.

[0033] (3) By combining the advantages of traditional deep learning and large models, the present invention can effectively improve the accuracy and robustness of biomedical named entity recognition and is suitable for entity recognition tasks in a multi-language environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a technical roadmap of an embodiment of the present invention.

[0035] Figure 2 This is a multi-corpus aggregation flowchart based on BERT-BiLSTM-CRF.

[0036] Figure 3 This is a flowchart of multi-corpus biomedical named entity recognition based on large model prompt word aggregation.

[0037] Figure 4 This is a flowchart of multi-corpus biomedical named entity recognition based on BERT-BiLSTM-CRF and large model corpus expansion. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0039] like Figure 1 As shown, the biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion described in the present invention includes two main parts: a multi-corpus aggregation method based on BERT-BiLSTM-CRF and a multi-corpus processing method based on large model corpus expansion.

[0040] Among them, the multi-corpus aggregation method based on BERT-BiLSTM-CRF mainly includes:

[0041] Data collection and preprocessing: Text data is collected from multiple public biomedical corpora (such as PubMed, BioNLP, ACLAnthology, etc.) and preprocessed, including removing irrelevant information (such as HTML tags and special symbols) and converting the data into BIO format (Beginning, Inside, Outside) to ensure that data from different corpora can be used uniformly.

[0042] Feature extraction and sequence modeling: Use the pre-trained BERT model to extract text features and capture contextual dependencies. Then use BiLSTM to further process sequence information and capture long-distance dependencies. Finally, use the conditional random field (CRF) layer to perform sequence annotation and optimize the accuracy of named entity recognition.

[0043] Multi-corpus joint training: Through multi-corpus joint training, the model can generalize on data from different corpora. To this end, the present invention designs a multi-standard joint loss function to balance the weights of different corpora and improve the training effect.

[0044] Model training and evaluation: The cross entropy loss function and dynamic learning rate adjustment strategy are used to optimize the model, and the model performance is evaluated through indicators such as F1 score, precision and recall.

[0045] The multi-corpus processing method based on large model corpus expansion mainly includes:

[0046] Prompt word mechanism and LoRA fine-tuning: Fine-tune large language models (such as Qwen2-7b, Llama3-8b, Gemma2-9b) through LoRA (Low-Rank Adaptation) technology. First, design prompt words to guide the model to generate corpus. The prompt words include task instructions and entity category restrictions, and can indicate the source of the corpus. For example: This sentence is from CorpusA's clinical information extraction. Please generate named entity recognition data in the biomedical field, with entities including onlyDisease, test, and treatment. The output should strictly follow this JSON format:{"text":"sentence","outputs":"entities in the sentence marked with <type> tags< / type> "}.

[0047] LoRA is used to fine-tune large models to reduce training parameters and optimize the model's ability to generate high-quality extended corpora under different prompt words and corpus conditions.

[0048] The large model generates extended corpus: The fine-tuned large model generates extended corpus based on the prompt words and outputs it in JSON format.

[0049] Processing and aggregation of the expanded corpus: The generated expanded corpus is preprocessed and formatted into BIO format, and re-aggregated with the original corpus to form the final training set.

[0050] Model training and entity recognition: After the expanded corpus is aggregated with the original corpus, it is input into the BERT-BiLSTM-CRF model for final training, and entity recognition is performed through the CRF layer. Through the joint training of the cross entropy loss function and the conditional random field, the log-likelihood function of the entity label sequence is maximized, thereby optimizing the named entity recognition performance.

[0051] Figure 2This shows the detailed operation process of multi-corpus aggregated biomedical named entity recognition based on BERT-BiLSTM-CRF. Figure 2 ,The core idea of ​​this embodiment is to improve the ,performance of biomedical named entity recognition (NER) by aggregating more ,annotated data. Therefore, the first task is to collect a large amount of ,text data in related fields that contain different types of entities. ,Ensuring the diversity and breadth of the dataset is the key to achieving good ,performance of the NER system.

[0052] In order to ensure the quality and consistency of the data, a series of preprocessing steps were performed on the collected data, including text cleaning, removal of unnecessary punctuation, HTML tags and special characters, and text standardization operations such as data format conversion to ensure the consistency and usability of the input data.

[0053] Subsequently, the corpus is integrated using the multi-corpus aggregation method, and finally it is input into the BERT-BiLSTM-CRF model for training and outputs the recognition results. Figure 2 shown.

[0054] S101. Data collection: In order to implement the named entity recognition method of multi-corpus aggregation, we designed a systematic corpus collection strategy, which includes search keywords, query criteria, filtering strategies and determination of candidate corpora.

[0055] S102. Keyword search and query criteria: Data collection mainly relies on scientific literature databases, including PubMed and ACL Anthology. In order to obtain multi-corpus data related to named entity recognition, we used a set of carefully selected keywords and query conditions for search. These keywords cover all aspects of the biomedical field, ensuring the breadth and representativeness of the data.

[0056] By searching databases such as PubMed using selected keywords, a large amount of literature was obtained, covering different topics in the biomedical field. The databases and resources retrieved included but were not limited to:

[0057] Biomedical Chemistry Databases: BC4CHEMD, BC5CDR.

[0058] Biomedical Natural Language Processing Competition Datasets: BioNLP09, BioNLP11, BioNLP13.

[0059] Disease named entity recognition datasets: NCBIDisease, AnatEM, BC2GM.

[0060] Other related datasets: CRAFT, JNLPBA, LINNAEUS, GENIA, ShARe / CLEFE, DDL, MedSTS, BIOSSES, ChemProt, i2b2-2010, Hoc, CoNLL-2003, SciERC, etc.

[0061] S103. Data selection criteria: In order to ensure the effectiveness of multi-language data aggregation and improve the quality of the dataset, we prioritize the following factors when selecting data:

[0062] Entity consistency: Ensure that entity categories in different corpora are consistent or comparable, so that the model can learn unified entity representations.

[0063] Entity type similarity: Select corpora with similar entity types to reduce the adverse effects of cross-corpus entity type differences on model training.

[0064] Balance between data quantity and quality: While ensuring data quality, expand the data scale as much as possible to improve the generalization ability of the model.

[0065] S104. Final candidate corpus: After the above screening and processing, we finally compiled a comprehensive and diverse candidate corpus, including:

[0066] BC5CDR: A labeled dataset containing chemicals and diseases.

[0067] BioNLP09, BioNLP11ID, BioNLP13CG, BioNLP13PC: datasets from multiple sessions of the Biomedical Natural Language Processing Competition.

[0068] NCBI-disease: An annotation dataset focusing on disease named entities.

[0069] CRAFT: A corpus of biomedical full-text annotated texts.

[0070] BC4CHEMD: A named entity recognition dataset for biomedical chemicals.

[0071] These corpora are obtained from PubMed and ACL Anthology, covering a variety of entity types in the biomedical field.

[0072] S105, text preprocessing: After data collection is completed, we preprocess the text data, mainly including:

[0073] Text cleaning: remove HTML tags, special characters and unnecessary punctuation marks from the text to ensure the purity of the text content.

[0074] Format conversion: Annotate the named entities in the text according to the BIO annotation format. The BIO format is a commonly used sequence annotation method that is used to identify whether each word is the beginning of an entity (B-label), the inside of an entity (I-label), or a non-entity (O). For example: Patient (O) was (O) discharged (O) to (O) rehabilitation (B-treatment) on (O) 2013-06-25 (O). (O)

[0075] Data standardization: Unify the data formats of different corpora to ensure data consistency and facilitate subsequent model training and evaluation.

[0076] S106. Data Aggregation: To efficiently aggregate multi-corpus data, we adopt a simple and practical approach based on manual labeling, inspired by the method proposed by Johnson et al.

[0077] Specifically, we add two manual tags at the beginning and end of the input sentence to specify the corpus source of the sentence. For example, for a sentence from BC5CDR, we add “ <corpusa> "and"< / corpusa> "mark.

[0078] These manual tags are treated as ordinary characters, specifying which dataset the sentence originated from. In this way, instances from different datasets can be seamlessly combined for joint training without the need for additional complex structures. The two special tags are designed to:

[0079] Carrying corpus source information: Through labeling, the model can identify the source of the sentence and thus learn the characteristics of different corpora.

[0080] Influencing contextual representations: Tag information can influence the contextual representation of each word, helping the model generate sequence annotations that match the target standard.

[0081] Simplified model structure: There is no need to introduce a complex encoder-decoder structure, and multi-corpus aggregation can be achieved through simple tagging.

[0082] During model training, these tags are fed into the model. During testing, the corresponding tags are also used to specify the desired segmentation criteria. The impact of these tags is not taken into account when calculating model performance.

[0083] S107, model training: input the preprocessed and aggregated data into the BERT-BiLSTM-CRF model for training. The specific steps are as follows:

[0084] BERT feature extraction: Use the pre-trained BERT model to encode the input text and obtain deep context feature representation. BERT can capture bidirectional context information and generate high-quality semantic representation for each word.

[0085] BiLSTM sequence modeling: Based on the feature representation of BERT, a bidirectional long short-term memory network (BiLSTM) is used to further model the sequence information and capture the long-distance dependencies in the sequence. BiLSTM can integrate the context information and improve the modeling ability of the sequence structure.

[0086] CRF sequence labeling: Finally, the conditional random field (CRF) layer is used to label the output of BiLSTM. The CRF layer takes into account the migration relationship between labels, which can globally optimize the sequence labeling results and improve the accuracy of named entity recognition.

[0087] Multi-corpus joint loss function: In model training, a multi-corpus joint loss function is defined to perform weighted optimization on data from different corpora.

[0088] Optimization and hyperparameter setting: Use the Adam optimizer for model training, and adopt dynamic learning rate adjustment strategies (such as Warmup and learning rate decay) to improve training stability and convergence speed. The model's hyperparameters (such as learning rate, batch size, weight decay, etc.) are optimized through methods such as cross-validation.

[0089] S108, Model evaluation: After the model training is completed, use an independent validation set to evaluate the model. The evaluation indicators include:

[0090] Precision: Measures how many of the positive examples predicted by the model are true positive examples.

[0091] Recall: Measures how many of the actual positive examples are correctly predicted by the model.

[0092] F1-score: The harmonic mean of precision and recall.

[0093] S109, Model Optimization: Based on the evaluation results, the model is iteratively optimized, including:

[0094] Adjust the model structure: adjust the number of layers and hidden units of BERT, and the number of layers and hidden dimensions of BiLSTM as needed.

[0095] Optimize hyperparameters: Through further experiments and verification, adjust hyperparameters such as learning rate and batch size to find the best configuration of the model.

[0096] Improve data processing: Depending on the performance of the model on a specific corpus, it may be necessary to re-examine the data preprocessing and aggregation methods to ensure data quality and consistency.

[0097] Figure 3 The detailed operation process of multi-corpus biomedical named entity recognition based on large model corpus expansion is illustrated. By using large pre-trained language models (such as Qwen2-7b, Llama3-8b, Gemma2-9b), combined with LoRA fine-tuning technology and prompt word mechanism, named entity recognition in a multi-corpus environment is achieved.

[0098] S201, collect biomedical text data from multiple public corpora. The data preprocessing step also includes text cleaning and format conversion.

[0099] S202, design prompt words for each corpus to help the model understand the source and task background of different corpora. The prompt words include task description, entity category restrictions, and optional corpus source identification.

[0100] Direct aggregation prompt words: The specific corpus source is not marked, only the task and entity category are specified. For example:

[0101] "instruction":"Please generate named entity recognition data in the biomedical field,with entities including only Disease,test,and treatment.Theoutput should strictly follow this JSON format:{\"text\":\"sentence\",\"outputs\":\"entities in the sentence marked with <type> tags< / type>\"}. The source label indicates the corpus the text is derived from. The example below is only to illustrate the output format, do not generate sentences that are identical or similar to the example. For example: {\"text\":\"On the evening of postoperative day number four, the patient had a transient episode of disorientation and confusion which subsequently resolved spontaneously.\",\"outputs\":\"On the evening of postoperative day number four, the patient had a transient episode of <disease> disorientation< / disease> and <disease> confusion< / disease> which subsequently resolved spontaneously.\"}"}

[0102] Prompt indicating the source: Clearly indicate the corpus source to help the model adjust the output. For example:

[0103] "instruction":"This sentence is from CorpusA's clinical information extraction. Please generate named entity recognition data in the biomedical field, with entities including only Disease, test, and treatment. The output should strictly follow this JSON format: {\"text\":\"sentence\",\"outputs\":\"entities in the sentence marked with <type> tags< / type>\"}.The source label indicates the corpus the text is derived from.The example below is only to illustrate the output format,do not generate sentences that are identical or similar to the example. <disease> disorientation< / disease> and <disease> confusion< / disease> which subsequently resolved spontaneously.\"}"}

[0104] S203. Fine-tune the large language model through LoRA technology to reduce model parameters and optimize its responsiveness to different prompt words. The formula is:

[0105] W adapted =W pretrained +α·AB

[0106] Among them, W adapted are the fine-tuned weights, A and B are low-rank matrices, and α is the scaling factor.

[0107] After fine-tuning, the model generates an extended corpus in BIO format through the prompt words. The generated extended corpus is merged with the original corpus as the training input of BERT-BiLSTM-CRF.

[0108] S204. During the process of fine-tuning the model and generating the expanded corpus, dynamically adjust and experiment with the prompt words to ensure that the prompt words can effectively guide the model to generate high-quality output. Through the prompt word ablation experiment, evaluate the impact of the prompt words on the model performance and find the optimal prompt word design.

[0109] S205: After the expanded corpus generated by the fine-tuned large model is aggregated with the original corpus, it is input into the BERT-BiLSTM-CRF model for training, and the model parameters are adjusted in combination with the results of the prompt word mechanism. Finally, the entity recognition task is performed in a multi-corpus environment.

[0110] S206. The model is evaluated through an independent validation set, using precision, recall and F1 score as the main indicators to ensure the adaptability and recognition performance of the model under different corpora.

[0111] S207. Based on the evaluation results, optimize the prompt word structure and apply data enhancement methods to further improve the generalization ability and performance of the model under different corpora.

[0112] Based on the aforementioned multi-corpus aggregation method and the multi-corpus processing method based on large model corpus expansion, an embodiment of the present invention discloses a biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion, the main steps of which include: collecting text data from multiple public biomedical corpora, performing text cleaning and annotating named entities in the text according to the BIO format; adding corpus source labels before and after each sentence to clearly indicate the corpus from which the sentence comes; expanding the corpus based on the large model through a prompt word mechanism, using prompt words to clearly explain the task and output the JSON format, and clearly annotating the source of each corpus in the prompt words to help the large model understand the characteristics of different corpora and generate expanded corpora with different corpus styles; after aggregating the expanded corpus with the BIO annotation format generated by the large model according to the prompt words with the original corpus, the expanded corpus is input into the BERT-BiLSTM-CRF model for final training, and biomedical named entity recognition is performed through the CRF layer, and the log-likelihood function of the entity label sequence is maximized through joint training of the cross entropy loss function and the conditional random field, so as to optimize the biomedical named entity recognition performance.

[0113] Figure 4 The detailed operation flow of the biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion is illustrated, and the steps include:

[0114] S301, data collection, first, collect biomedical related text data from multiple public biomedical corpora (such as PubMed, ACL Anthology, BioNLP, etc.). Each corpus contains different categories of named entities such as diseases, drugs, genes, etc. These entities form the basis of corpus annotation.

[0115] S302: After data collection is completed, perform systematic preprocessing operations on the original text data to improve data quality and model training effects. Specifically, it includes:

[0116] Text cleaning: Remove useless elements from the text, such as HTML tags, special symbols, and characters that are not relevant to semantics (such as redundant punctuation marks). This step ensures the purity of the text and reduces the interference of noise on model training.

[0117] Unified annotation format: Annotate the named entities in the text according to the BIO format (Beginning, Inside, Outside). The BIO format is a commonly used sequence annotation method that clearly identifies whether each word is the beginning of an entity (B-label), the inside of an entity (I-label), or a non-entity (O). Through a unified annotation format, data from different corpora can be aggregated together to form a consistent model input.

[0118] S303: Based on the pre-processed data, multi-corpus aggregation processing is performed, mainly using the following two methods, among which the aggregation method that specifies the source of the corpus is the method adopted by the present invention, and the direct aggregation method is used for effect comparison:

[0119] Direct aggregation: directly merge the BIO format training sets of multiple corpora to form a comprehensive training dataset. This method is simple and direct, but it may ignore the characteristic differences between different corpora.

[0120] Aggregation that indicates the source of the corpus: Add source tags before and after each sentence to clearly indicate which corpus the sentence comes from. For example: <corpusa> Patient(O)was(O)discharged(O)to(O)rehabilitation(B-treatment)on(O)2013-06-25(O).(O)< / corpusa> This approach helps the model identify the source of the corpus. These labels are embedded into the context modeling when passing through the BERT layer and the BiLSTM layer, allowing the model to distinguish the characteristics of different corpora and make corresponding context adjustments based on the source of the corpus. Its role is similar to task encoding in multi-task learning, ensuring that the model can optimize predictions based on different corpus sources.

[0121] In the BERT-BiLSTM-CRF model, adding clear corpus source labels has an important impact on the model's context modeling and sequence label prediction. Through clear label instructions, the model can make better context adjustments when processing different corpora, and ultimately improve the accuracy of named entity recognition. This is specifically reflected in the following aspects:

[0122] 1. Modeling of contextual features

[0123] When adding source tags before and after the input sequence (such as <coupusa> and< / coupusa>), these labels are jointly involved in the modeling process of contextual features through the BERT layer and the BiLSTM layer. BERT is responsible for capturing the global semantic information of the sentence, while BiLSTM serializes the context of each time step. As part of the input, the artificial label affects the contextual features h generated at each time step. t , so that the BiLSTM layer can generate different feature representations according to different corpus sources. The formula is:

[0124] h t =BiLSTM(E,t)

[0125] Among them, E is the input embedding representation containing the source label of the corpus. By adding labels, the model can use these label prompts to learn the characteristics of different corpora, ensuring that the model makes corresponding adjustments when processing different corpora.

[0126] 2. Inference of Global Label Sequence

[0127] In the CRF (Conditional Random Field) layer, the model infers the global label sequence through context features and uses negative log-likelihood loss to optimize the model. The loss function is defined as follows:

[0128] L=-logP(Y|X)

[0129] Among them, P(Y|X) is the conditional probability of the correct label sequence Y given the input sequence X. The CRF layer predicts the optimal path of the sequence by calculating the score s(X,Y) of the sequence. The specific scoring formula is:

[0130]

[0131] A is the transfer score matrix between labels, indicating the transition from label y i to y i+1 transfer score. is the feature representation of the i-th character and the label y i score.

[0132] By adding the corpus source label, the model generates contextual features h in the BiLSTM layer t , these features directly affect the score function of the CRF layer and the transfer matrix A. This means that the impact of manual labels on context modeling is passed on to the calculation of label scores, helping the model adjust the predicted labels based on different corpus sources.

[0133] 3. The impact of manual labeling on the loss function

[0134] Manual labeling affects the context feature h t, which in turn affects the scoring calculation in the CRF layer. For different corpora, the model will generate different context representations based on different tags, which will affect the transition probability and score of the label.

[0135] Contextual consistency: The presence of manual labeling helps the model generate consistent contextual features across different corpora, enabling the model to perform word segmentation and named entity recognition based on the style and structure of a specific corpus.

[0136] Shared knowledge and adjustment: Although the model parameters are shared in multi-corpus learning, the manual labeling provides additional distinguishing information, so the model can fine-tune the features of different corpora based on the shared parameters. In this way, under different tasks, the model's loss calculation can better align the scores of the correct path and improve the final prediction effect.

[0137] By adding corpus source labels, the model can improve named entity recognition in the following ways:

[0138] Clear task instructions: Labels help the model clearly distinguish between different corpus sources and ensure that the score of the correct path in the loss function can reflect the characteristics of the specific corpus.

[0139] Contextual adjustment: labels are adjusted by influencing the context feature h t , ensuring that the label prediction of the CRF layer can adapt to different corpus standards, thereby improving the accuracy of word segmentation and named entity recognition.

[0140] Knowledge sharing and differentiation: The introduction of labels improves the model’s ability to fine-tune specific corpora under shared parameters.

[0141] S304. Based on a large language model (such as Qwen2-7b, Llama3-8b, Gemma2-9b), the corpus is expanded through the prompting mechanism. The large language model can understand the instructions in the prompt words and generate new high-quality corpus based on the existing biomedical corpus. The specific process is as follows:

[0142] Prompt word design: Prompt word design needs to clarify the task type, entity category, and output format. Specifically, first, clarify the task goal, tell the model the source of the sentence, and require the model to generate new named entity recognition data, indicating the entity category in the task; then, provide specific requirements for input and output, and ensure that the output strictly follows the following JSON format: {"text":"sentence","outputs":"entities in the sentence marked with <type> tags< / type>"}; At the same time, it is required to indicate the source of the text; To avoid generating duplicate or similar content, it is clearly stated that the examples are only used to illustrate the output format; Finally, examples are used to show how to use labels to annotate entity categories in the output.

[0143] Taking direct aggregation as an example, the prompt includes a clear task description and the output JSON format. Prompt instruction example: "instruction": "Please generate named entity recognition data in the biomedical field, with entities including only Disease, test, and treatment. The output should strictly follow this JSON format: {\"text\":\"sentence\",\"outputs\":\"entities in the sentence marked with <type> tags< / type> \"}.The example below is only to illustrate the output format,do not generatesentences that are identical or similar to the example. <disease> disorientation< / disease> and <disease> confusion< / disease> which subsequently resolved spontaneously.\"}"

[0144] Prompt word processing:

[0145] Direct aggregation method: The model is instructed to expand the corpus through prompt words. The prompt words clearly state that the task is named entity recognition (NER) and the entity type is clear.

[0146] The method of indicating the source of the corpus: similar to direct aggregation, but the source of each corpus needs to be clearly marked in the prompt word. This can help the large model understand the characteristics of different corpora and generate extended corpora with different corpus styles. For example: {

[0147] "instruction":"This sentence is from CorpusA's clinical informationextraction. Please generate named entity recognition data in the biomedicalfield,with entities including only Disease,test,and treatment.The output should strictly follow this JSON format:{\"text\":\"sentence\",\"outputs\":\"entities in the sentence marked with <type> tags< / type> \"}.The source label indicates the corpus the text is derived from.The example below is only to illustrate the output format,do not generate sentences that are identical or similar to the example.<disease> disorientation< / disease> and <disease> confusion< / disease> which subsequently resolved spontaneously.\"}"}. Through the prompt words, the model can automatically generate new corpus and expand the amount of training data. The prompt word mechanism not only makes the generated corpus conform to the characteristics of the original corpus, but also enriches the diversity of named entities. Clearly marking the source of the corpus helps to further optimize the model's adaptability to different corpus styles.

[0148] Furthermore, before generating the extended corpus, the large model is first fine-tuned and trained using the LoRA (Low-Rank Adaptation) technology, so that the model can better adapt to the original corpus generation process guided by the prompt words. Through LoRA fine-tuning, the model can efficiently generate extended corpora consistent with the characteristics of different corpora. The application of LoRA technology can significantly reduce the number of parameters required for model fine-tuning while maintaining its performance in generating high-quality corpora. The specific steps are as follows:

[0149] LoRA Introduction: LoRA is a low-rank adaptive fine-tuning technology. Its main principle is to decompose the original model weight matrix to form two low-rank matrices A and B, and fine-tune the model parameters by adjusting these two matrices. The formula is as follows:

[0150] W adapted =W pretrained +α·AB

[0151] Among them, W adpted represents the fine-tuned weight matrix, W pretrained is the weight of the pre-trained model, α is the scaling factor, and A and B are low-rank matrices.

[0152] Before corpus expansion, LoRA is used to fine-tune the large model to ensure that the model can quickly adapt to changes in the generation of different prompt words. This process can be expressed by the formula:

[0153]

[0154] Among them, W t is the model weight of the tth iteration, η is the learning rate, is the gradient based on the prompt word P and the input X.

[0155] The main goal of LoRA fine-tuning is to reduce parameter overhead and ensure that the model can efficiently handle multiple prompt words and corpus sources, and ultimately optimize the quality of the generated extended corpus. The fine-tuned large model can better understand the prompt words and generate corpora that conform to the style of different corpora.

[0156] During the LoRA fine-tuning process, the original corpus and prompt words are used to guide model training, with the goal of optimizing model weights. The loss function during training uses cross entropy loss, and the formula is as follows:

[0157]

[0158] Among them, L represents the cross entropy loss, which is used to calculate the deviation of the model from the true label when predicting the entity type of each word or phrase, N represents the total number of samples involved in the calculation of the cross entropy loss, and y i is the true label, corresponding to the correct entity type annotation for each word or phrase in the text, The label probability predicted by the model indicates the model's prediction result for the entity type of each word or phrase, and gives the model's confidence in each possible category in the form of probability.

[0159] The training process is optimized using dynamic learning rate adjustment strategies such as warmup and learning rate decay to ensure training stability and convergence.

[0160] After the fine-tuning of LoRA is completed, the fine-tuned large model is used to generate the extended corpus through the prompt words. The generation of the extended corpus can be expressed as:

[0161] Y augmented =Model LoRA (X,P)

[0162] Among them, Y augmented This is the expanded corpus generated by the large model fine-tuned by LoRA.

[0163] S305, large model generates corpus: after receiving the prompt word, the model will generate new corpus according to the prompt and output it in JSON format.

[0164] "text":"On the evening of postoperative day number four, the patient had a transient episode of disorientation and confusion which subsequently resolved spontaneously.",

[0165] "outputs":"On the evening of postoperative day number four, the patient had a transient episode of <disease> disorientation< / disease> and <disease> confusion< / disease>which subsequently resolved spontaneously."

[0166] }

[0167] After the augmented corpus is generated, it is first converted into BIO annotation form and then aggregated with the original corpus. At this point, the final corpus generated will be used for the subsequent training of the BERT-BiLSTM-CRF model to further improve the accuracy of named entity recognition.

[0168] S306, BERT feature extraction, using the BERT (Bidirectional Encoder Representations from Transformers) model as the basis for text feature extraction. BERT can learn contextual dependencies in text and generate deep semantic feature representations through the bidirectional Transformer structure. The input of BERT is a vector representation composed of three parts: Token Embedding, Segment Embedding, and Position Embedding:

[0169] E=E+E+E

[0170] Token segment position

[0171] Among them, E is the embedding representation of the input text, E token is word embedding, E segment is the sentence fragment embedding, E position Embed for position.

[0172] BERT calculates context-dependent representations through Transformer’s Multi-head Attention mechanism:

[0173]

[0174] Where Q, K, and V represent query, key, and value matrices respectively. k is a scaling factor to prevent the dot product from being too large.

[0175] Through the multi-layer Transformer structure, BERT can capture the global semantic information of the input sequence and generate context-aware feature representations for each word.

[0176] S307. Based on the feature representation generated by BERT, the sequence is further processed using a bidirectional long short-term memory network (BiLSTM). BiLSTM can capture long-distance dependency information in the text and generate semantic representations in two directions through forward and reverse LSTM units.

[0177] Given an input sequence x = [x1, x2, ..., x T ], BiLSTM calculates the hidden state representation of each time step:

[0178]

[0179] in, and are the hidden states of the forward and reverse LSTMs, respectively, and are combined to generate the final output representation of the BiLSTM h t .

[0180] S308: In order to enhance the model's ability to model the dependency of label sequences, a conditional random field (CRF) is applied to annotate the final features. CRF can capture the dependency between labels in a sequence, thereby improving the accuracy of named entity recognition.

[0181] Given the feature representation of BiLSTM output H = [h1,h2,...,h T ], CRF converts the label prediction of each time step into a serialized labeling task by learning the transfer matrix A of sequence labeling. The goal of the CRF layer is to maximize the probability of the correct label sequence:

[0182]

[0183] in, Represents the transfer matrix between labels (from label y t-1 Move to label y t The transition matrix A contains the transition probabilities between all possible labels, indicating the dependency between labels in the sequence. Represents the label y t The corresponding classification weight. W is used to calculate the label y of the current time step t t The hidden state h corresponding to this time step t The score of h t is the context representation vector output by BiLSTM at time step t, which contains the context features of the current time step. T represents the length of the sequence. Y represents the set of all possible label sequences.

[0184] The CRF layer is trained by maximizing the log-likelihood of the correct sequence:

[0185] L CRF = -logP(y|H)

[0186] Where L CRF is the loss of the model on the CRF layer, calculated by negative log likelihood. P(y|H) represents the probability of the subscript sequence y given the input feature H.

[0187] S309: In order to better fit the multi-corpus aggregation training with manual labels, a multi-corpus joint loss function is innovatively designed. This loss function introduces the weights α of different corpora. i , in order to balance the impact of each corpus on model training. The formula of the loss function is as follows:

[0188]

[0189] Where N is the number of corpora, α i is the weight of the i-th corpus, L i is the loss function of the i-th corpus, P i (y|x) is the predicted probability of the i-th corpus, y i is the true label of the i-th corpus.

[0190] This loss function performs a weighted summation of the losses of each corpus to ensure that different corpora have a balanced impact on the training of the model.

[0191] Weight α i It can be set through prior knowledge (such as the size or quality of the corpus) or dynamically adjusted during training to adapt to the characteristics of different corpora.

[0192] In the BERT-BiLSTM-CRF model, when calculating the loss of each corpus, there is no need to change the basic structure of the model. The features of each corpus are extracted by BERT, the BiLSTM performs sequence encoding, and the CRF layer processes the dependencies between label sequences.

[0193] S310, using the Adam optimizer, uses dynamic learning rate adjustment strategies (such as Warmup and learning rate decay) to improve the stability and effect of training. The dynamic learning rate strategy is defined as:

[0194]

[0195] Where η0 is the initial learning rate, t warmupt is the number of steps in the Warmup phase.

[0196] S311. Evaluate the performance of the model on an independent validation set, and measure the entity recognition ability of the model through F1 score, precision, and recall. The calculation formula for F1 score is:

[0197]

[0198] Among them, precision and recall are defined as:

[0199]

[0200] Among them, TP is a true positive example, FP is a false positive example, and FN is a false negative example.

[0201] S312. Perform iterative optimization based on the evaluation results of the model on the validation set, including:

[0202] Adjust the model structure: According to actual needs, adjust the number of layers and hidden units of BERT, and the number of layers and hidden dimensions of BiLSTM to improve the expressiveness of the model.

[0203] Optimize hyperparameters: Use grid search or Bayesian optimization to adjust hyperparameters such as learning rate, batch size, weight decay, etc. to find the optimal configuration of the model.

[0204] Regularization strategy: Introduce methods such as Dropout and L2 regularization to prevent model overfitting and improve the generalization ability of the model.

[0205] Early stopping strategy: When the performance of the validation set no longer improves, terminate the training early to avoid overfitting the model to the training data.

[0206] Data enhancement: Use data enhancement techniques, such as synonym replacement and random deletion, to increase the diversity of training data and improve the model's adaptability to unknown data.

[0207] The above steps are iterated repeatedly until the model achieves the best performance on the validation set.

[0208] Table 1 shows the experimental results of comparing direct aggregation of multiple corpora with aggregation schemes that specify the corpus source (without expanding the corpus with a large model). Bold indicates better results than those in the SciBERT-Base Vocab paper, and + indicates the performance improvement of the scheme that specifies the corpus source compared to the results in the SciBERT-Base Vocab paper. F1(tokens) indicates the result of using manual tags to specify the corpus source, and F1(No-tokens) indicates the result of direct aggregation without manual tags. The abbreviations 11ID, 13CG, 13PC, BC5, BC4, and CR represent the datasets BioNLP11ID, BioNLP13CG, BioNLP13PC, BC5CDR, BC4CHEMD, and CRAFT, respectively. Test data: BC5CDR.

[0209] Table 1 Comparative experimental results of multi-corpus direct aggregation and corpus source aggregation schemes

[0210]

[0211]

[0212] Table 2 shows the experimental results of comparing direct aggregation of multiple corpora with aggregation schemes that specify the corpus source (using large models to expand the corpus). Bold indicates better results than specifying the corpus source (without expanding the corpus with large models), and + indicates the performance improvement of using large models to expand the corpus compared to not expanding the corpus with large models. F1(tokens) indicates the results of aggregation using manual tags to specify the corpus source, and F1(No-tokens) indicates the results of direct aggregation without manual tags. The abbreviations 11ID, 13CG, 13PC, BC5, and CR represent the datasets BioNLP11ID, BioNLP13CG, BioNLP13PC, BC5CDR, and CRAFT, respectively. Test data: BC5CDR.

[0213] Table 2 Comparative experimental results using large model to expand corpus

[0214]

[0215] Based on the same inventive concept, the present invention provides a biomedical named entity recognition system based on multi-corpus aggregation and large model corpus expansion, comprising:

[0216] The multi-corpus aggregation module based on BERT-BiLSTM-CRF is used to collect text data from multiple public biomedical corpora, perform text cleaning and annotate named entities in the text according to the BIO format; add corpus source labels before and after each sentence to clearly indicate the corpus from which the sentence comes; and aggregate the expanded corpus generated by the corpus expansion module based on the large language model with the original corpus, input it into the BERT-BiLSTM-CRF model for final training, and perform biomedical named entity recognition through the CRF layer. Through the joint training of the cross entropy loss function and the conditional random field, the log-likelihood function of the entity label sequence is maximized to optimize the biomedical named entity recognition performance;

[0217] And the corpus expansion module based on large language models is used to expand the corpus based on the large model through the prompt word mechanism. The prompt words clearly explain the task and output the JSON format. At the same time, the source of each corpus is clearly marked in the prompt words, which helps the large model understand the characteristics of different corpora and generate expanded corpora with different corpus styles.

[0218] The present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion.

Claims

1. A biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion, characterized in that: The steps include: Collect text data from multiple public biomedical corpora, perform text cleaning and annotate named entities in the text according to BIO format; Add source tags before and after each sentence to clearly indicate the corpus the sentence comes from; The corpus is expanded based on the big model through the prompt word mechanism. The prompt words clearly indicate the task description and the output JSON format. At the same time, the source of each corpus is clearly marked in the prompt words, which helps the big model understand the characteristics of different corpora and generate expanded corpora with different corpus styles. After the extended corpus with BIO annotation format generated by the large model according to the prompt words is aggregated with the original corpus, it is input into the BERT-BiLSTM-CRF model for final training, and biomedical named entity recognition is performed through the CRF layer. Through the joint training of the cross entropy loss function and the conditional random field, the log-likelihood function of the entity label sequence is maximized, thereby optimizing the biomedical named entity recognition performance.

2. The biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion according to claim 1 is characterized in that: The prompt words for expanding the corpus through the large model are designed as follows: first, clarify the task goal, tell the model the source of the sentence, and require the model to generate new named entity recognition data, indicating the entity category in the task; then, provide specific requirements for input and output, and ensure that the output strictly follows the following JSON format: {"text":"sentence","outputs":"entities in the sentence marked with <type> tags< / type> "}; At the same time, it is required to indicate the source of the text; To avoid generating duplicate or similar content, it is clearly stated that the examples are only used to illustrate the output format; Finally, examples are used to show how to use labels to annotate entity categories in the output.

3. The biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion according to claim 1 is characterized in that: Before generating the extended corpus, the large model is fine-tuned and trained using the LoRA technology, so that the model can better adapt to the original corpus generation process guided by prompt words.

4. The biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion according to claim 3 is characterized in that: During the LoRA fine-tuning large model, the loss function uses cross entropy loss, and the loss function is: Where L represents the cross entropy loss, which is used to calculate the deviation of the model from the true label when predicting the entity type of each word or phrase. N represents the total number of samples involved in the calculation of the cross entropy loss. i is the true label, corresponding to the correct entity type annotation for each word or phrase in the text, The label probability predicted by the model indicates the model's prediction result for the entity type of each word or phrase, and gives the model's confidence in each possible category in the form of probability.

5. The biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion according to claim 1 is characterized in that: In the BERT-BiLSTM-CRF model, the BERT model is used as the basis for text feature extraction. BERT learns the contextual dependencies in the text through a bidirectional Transformer structure and generates deep semantic feature representations. Based on the feature representation generated by BERT, a bidirectional long short-term memory network BiLSTM is used to further process the sequence, and semantic representations in two directions are generated through forward and reverse LSTM units. The conditional random field CRF is used to annotate the final features.

6. The biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion according to claim 1 is characterized in that: When training the BERT-BiLSTM-CRF model, a multi-corpus joint loss function is defined to perform weighted optimization on the data of different corpora. The joint loss function is: Where N is the number of corpora, α i is the weight of the i-th corpus, L i is the loss function of the i-th corpus, P i (y|x) is the predicted probability of the i-th corpus, y i is the true label of the i-th corpus.

7. The biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion according to claim 1 is characterized in that: In the process of fine-tuning the large model and generating expanded corpus, the prompt words are dynamically adjusted and experimented. Through the prompt word ablation experiment, the impact of the prompt words on the model performance is evaluated to find the optimal prompt word design.

8. The biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion according to claim 1 is characterized in that: After the BERT-BiLSTM-CRF model is trained, the model performance is evaluated on an independent validation set. The F1 score, precision, and recall are used to measure the entity recognition capability of the model. Based on the evaluation results of the model on the validation set, iterative optimization is performed, including: Adjust the model structure: adjust the number of layers and hidden units of BERT, and the number of layers and hidden dimensions of BiLSTM to improve the expressiveness of the model; Optimize hyperparameters: Use grid search or Bayesian optimization methods to adjust learning rate, batch size, weight decay, and find the optimal configuration of the model; Regularization strategy: Introduce Dropout and L2 regularization methods to prevent model overfitting and improve the generalization ability of the model; Early stopping strategy: When the performance of the validation set no longer improves, the training is terminated early to avoid the model overfitting the training data; Data enhancement: Use data enhancement technology to increase the diversity of training data and improve the model's adaptability to unknown data.

9. A biomedical named entity recognition system based on multi-corpus aggregation and large model corpus expansion, characterized in that: include: A multi-corpus aggregation module based on BERT-BiLSTM-CRF, which is used to collect text data from multiple public biomedical corpora, perform text cleaning, and annotate named entities in the text according to the BIO format; The corpus source label is added before and after each sentence to clearly indicate the corpus from which the sentence comes; the expanded corpus generated by the corpus expansion module based on the large language model is aggregated with the original corpus, and then input into the BERT-BiLSTM-CRF model for final training, and biomedical named entity recognition is performed through the CRF layer. Through the joint training of the cross entropy loss function and the conditional random field, the log-likelihood function of the entity label sequence is maximized, and the biomedical named entity recognition performance is optimized; And the corpus expansion module based on large language models is used to expand the corpus based on the large model through the prompt word mechanism. The prompt words clearly explain the task and output the JSON format. At the same time, the source of each corpus is clearly marked in the prompt words, which helps the large model understand the characteristics of different corpora and generate expanded corpora with different corpus styles.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the biomedical named entity recognition method based on multi-corpus aggregation and large model corpus expansion are implemented according to any one of claims 1 to 8.

Images