Method and system for automatically extracting entity relationship in clinical trial qualification standard based on multi-round questions and answers

Through the combination of multiple rounds of question-and-answer methods and the BERT-BiLSTM-CRF model, the problems of low recall and error accumulation in the extraction of clinical trial qualification standards were solved, and high-precision entity relationship extraction was achieved, which improved the accuracy of the extraction results.

CN118917307AActive Publication Date: 2024-11-08ZHEJIANG UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411064912.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-08
Estimated Expiration
2044-08-05

AI Technical Summary

Technical Problem

The recall rate of the information extraction method for the existing clinical trial qualification standard is low, and the assembly line method has error accumulation problems, resulting in deviations in the extraction results.

Method used

Using a multi-round question-and-answer method, the head and tail body questions are input into the BERT-BiLSTM-CRF model, and the high-precision entity relationship automatic extraction of clinical trial qualification standards is achieved through multiple rounds of question-and-answer. The power of the BERT-BiLSTM-CRF model is used. Semantic comprehension ability, combined with multiple rounds of question-and-answer paradigms, reduce error accumulation.

Benefits of technology

This improves the recall rate of the model, realizes high-precision entity relationship extraction of the qualification standards of complex clinical trials, enhances semantic understanding ability, reduces error accumulation, and improves the accuracy of the extraction results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917307A_ABST
    Figure CN118917307A_ABST
Patent Text Reader

Abstract

The invention discloses a method and a system for automatically extracting an entity relationship in a clinical trial qualification standard based on multi-round questions and answers. The method comprises the following steps of: preprocessing an obtained clinical trial qualification standard text to obtain a free text; defining an entity category and a relationship category, labeling the free text, and designing a head entity problem template and a tail entity problem template; constructing positive samples and negative samples of head entity questions and answers and tail entity questions and answers to train the BERT-BiLSTM-CRF model; and splicing the head entity question and the free text, inputting the spliced head entity question and the free text into the trained model, predicting to obtain a head entity name, replacing the head entity name into the tail entity question, splicing the tail entity question and the free text again, inputting the spliced head entity question and the free text into the model, predicting to obtain a tail entity name, and finally predicting all entities in the free text and relationships thereof by the model through multiple rounds of questions and answers. According to the method, high-precision entity relationship automatic extraction in a clinical scene of a complex clinical test qualification standard can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of medical text mining, and in particular relates to a method and system for automatically extracting entity relationships in clinical trial qualification standards based on multi-round question-answering. Background Art

[0002] Clinical trials provide patients with new hope for treatment when traditional treatment options are ineffective. Clinical trial websites allow doctors to search for suitable trial items by keywords, but due to the limitations of keywords, search results are often lengthy, the screening process is time-consuming and inefficient, and the semantic complexity of eligibility criteria makes manual review cumbersome and labor-intensive. Therefore, the study of automatic extraction methods is particularly important.

[0003] In the early days, the automatic extraction of eligibility criteria mainly adopted rule-based and dictionary-based methods. The core of this method is to use predefined rules and vocabularies to identify entities in the text and the relationships between them. Due to its simple implementation and rapid execution, it was applied in the early days. However, this method also has obvious limitations: rules and dictionaries need to be manually updated to adapt to new entities or relationships, and it is difficult to cover all expressions and language diversity in the text. Therefore, when faced with complex and changeable natural languages, this method requires a lot of manual work and its effectiveness is often limited.

[0004] With the rapid development of machine learning and deep learning technologies, a variety of effective methods have been proposed to accurately identify and extract eligibility criteria in clinical trial information extraction.

[0005] The Criteria2Query system (Yuan C, Ryan PB, Ta C, et al (2019) Criteria2Query: a natural language interface to clinical databases for cohort definition. Journal of the American Medical Informatics Association, 26 (4), 294-305.) is an early attempt at machine learning in this field. It has achieved excellent performance in entity recognition and relationship extraction through steps such as sentence segmentation, named entity recognition, relationship extraction and logical detection.

[0006] Tseo Y et al. further explored the application of deep neural networks in information extraction of eligibility criteria and proposed the Att-BiLSTM-CRF model (Tseo Y, Salkola MI, Mohamed A, et al (2020) Information extraction of clinical trial eligibility criteria.arXiv preprint arXiv:2006.07296.). By introducing attention mechanism and conditional random field, the model showed comparable or even better performance than Criteria2Query in entity recognition and relationship extraction.

[0007] Furthermore, the emergence of the pre-trained language model BERT has brought new opportunities to the field of information extraction. The CT-BERT model developed by Liu X et al. (Liu X, Hersch GL, Khalil I, et al (2021) Clinical trial information extraction with BERT. 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI). IEEE: p. 505-506) takes advantage of BERT's powerful text representation capabilities and achieves efficient extraction of entities and relations by fine-tuning the pre-trained model, showing performance that exceeds the previous methods, demonstrating the potential of pre-trained language models in information extraction tasks.

[0008] Although these methods have achieved impressive results in terms of precision, they generally face the problem of relatively low recall rates. Due to the complexity of clinical trial eligibility criteria, different semantic categories and patterns, and multiple patterns of standards, these texts usually contain multiple entity types and complex relationships, and usually use professional terms and mixed English-Chinese expressions, which requires a high level of semantic understanding of the model. In addition, current research on eligibility criteria information extraction relies on a pipeline approach, which has inherent defects in error accumulation and also leads to low recall rates. Errors in each subtask may lead to omissions of key entities or errors in relationship recognition. These accumulated errors will bias the final extraction results, thereby negatively affecting the system's recall rate. Summary of the invention

[0009] In view of the above, the purpose of the present invention is to provide a method and system for automatic extraction of entity relationships in clinical trial qualification standards based on multi-round question and answering, by inputting head entity questions and tail entity questions into a BERT-BiLSTM-CRF model for multi-round question and answering, thereby realizing high-precision automatic extraction of entity relationships of clinical trial qualification standards in clinical scenarios with complex clinical trial qualification standards.

[0010] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0011] In a first aspect, an embodiment of the present invention provides a method for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question-answering, comprising the following steps:

[0012] The obtained clinical trial eligibility criteria text is preprocessed including sentence segmentation and cleaning to obtain a free text of the clinical trial eligibility criteria;

[0013] Based on the free text, the entity categories and relationship categories are defined, the free text is annotated according to the defined entity categories and relationship categories to obtain annotated data, and a head entity question template and a tail entity question template are designed for the head entity category and the tail entity category in the entity category respectively;

[0014] Based on free text, annotated data, and question templates, construct positive samples of head entity question answering and tail entity question answering. Based on free text and unannotated entity categories, construct negative samples. Input positive and negative samples into the BERT-BiLSTM-CRF model for model training.

[0015] In each round of question-answering, the head entity question is concatenated with the free text and input into the trained BERT-BiLSTM-CRF model to predict the head entity name. The head entity name is then replaced into the corresponding tail entity question and concatenated with the free text again and input into the model to predict the tail entity name. Through multiple rounds of question-answering, the model can eventually predict all entities and their relationships in the free text.

[0016] Specifically, the entity categories and relationship categories defined based on free text include:

[0017] Entity words or phrases in free text are analyzed and a head entity category and a tail entity category, as well as a relationship category between the head entity category and the tail entity category, are defined, wherein the head entity category is the entity category as the starting point of the relationship, and the tail entity category is the entity category as the end point of the relationship.

[0018] Specifically, the step of annotating the free text according to the defined entity categories and relationship categories to obtain annotated text data includes:

[0019] Using the brat tool, the entities in the free text are annotated with entity categories and relationship categories according to the entity categories and relationship categories and the established annotation guidelines, and the annotated text data corresponding to each free text is obtained.

[0020] Specifically, the designing of a head entity question template and a tail entity question template for the head entity category and the tail entity category in the entity category respectively includes:

[0021] Simulating the way humans ask questions, a head entity question template is designed according to the head entity category, and a tail entity question template that lacks a subject is designed according to the tail entity category. When asking and answering questions, the answer to the head entity question is added as the subject to the tail entity question template to form a tail entity question.

[0022] Specifically, the positive samples of constructing head entity question and answer and tail entity question and answer based on free text, annotated data and question template include:

[0023] For a single head entity category, the corresponding head entity question is concatenated with the free text to construct a positive head entity question input and a positive answer true label;

[0024] For the head and tail entity category pairs, the head entity question corresponding to the head entity category is concatenated with the free text to form a positive head entity question input and the corresponding positive answer true label is given. The answer to the head entity question is added to the tail entity question template corresponding to the tail entity category to form a tail entity question. The tail entity question is concatenated with the free text to form a positive tail entity question input and the corresponding positive answer true label is given.

[0025] Specifically, the negative samples are constructed based on free text and unlabeled entity categories, including:

[0026] Select the head entity categories that do not appear in the free text and are therefore not labeled, concatenate the corresponding head entity questions with the free text to construct a negative head entity question input and give the corresponding negative answer empty label;

[0027] Select the tail entity category that does not appear in the free text and is therefore not labeled, replace the labeled head entity name into the tail entity question template to form a tail entity question, concatenate the tail entity question with the free text to construct a negative tail entity question input and give the corresponding negative answer empty label.

[0028] Specifically, through multiple rounds of question-answering, the model finally predicts all entities and their relationships in the free text, including:

[0029] In multiple rounds of question answering, the model recognizes all head entity names in free text by iterating different head entity questions; for each head entity name, it is replaced with the subject part in the tail entity question, and the model recognizes all tail entity names and corresponding relationship categories that have a relationship with the head entity name in the free text by iterating different tail entity questions; if the entity category corresponding to the question does not exist in the free text, the model will output no relevant entity.

[0030] In a second aspect, to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a system for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question-answering, comprising: a data acquisition and preprocessing module, a model training preparation module, a model training module and an entity relationship extraction prediction module;

[0031] The data acquisition and preprocessing module is used to perform preprocessing including sentence segmentation and cleaning on the acquired clinical trial qualification standard text to obtain a free text of the clinical trial qualification standard;

[0032] The model training preparation module is used to define entity categories and relationship categories in the free text based on the free text, annotate the free text according to the defined entity categories and relationship categories to obtain annotated data, and design a head entity question template and a tail entity question template for the head entity category and the tail entity category in the entity category respectively;

[0033] The model training module is used to construct positive samples of head entity question and answer and tail entity question and answer based on free text, annotated data and question templates, construct negative samples based on free text and unannotated entity categories, and input the positive samples and negative samples into the BERT-BiLSTM-CRF model for model training;

[0034] The entity relationship extraction prediction module is used to splice the head entity question with the free text in each round of question and answer, and then input the spliced ​​head entity question into the trained BERT-BiLSTM-CRF model to predict the head entity name, and then replace the head entity name into the corresponding tail entity question and splice it with the free text again and input the model to predict the tail entity name. Through multiple rounds of question and answer, the model finally predicts all entities and their relationships in the free text.

[0035] In the third aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides a device for automatically extracting entity relationships in clinical trial qualification standards based on multi-round question and answer, including a memory and one or more processors, the memory is used to store computer programs, and the processor is used to implement the above-mentioned method for automatically extracting entity relationships in clinical trial qualification standards based on multi-round question and answer when executing the computer program.

[0036] In a fourth aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned method for automatically extracting entity relationships in the clinical trial qualification criteria based on multi-round questions and answers is implemented.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) The present invention can comprehensively cover the category standards of different contents described in the qualification criteria by preprocessing the qualification criteria text and deeply analyzing the contents therein. By defining the head entity category and the tail entity category and their relationship categories and annotating the free text, the corresponding head entity questions and tail entity questions are tailored for the multi-round question-answering paradigm, and high-precision automatic extraction of entity relationships is achieved through multi-round question-answering of the model.

[0039] (2) The present invention seamlessly integrates the multi-round question-answering paradigm with the BERT-BiLSTM-CRF model, integrating and giving full play to the high semantic understanding capabilities of the BERT-BiLSTM-CRF model and the multi-round question-answering paradigm, while covering prior knowledge in entity questions, effectively alleviating the error accumulation problem and improving the recall rate of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 It is a flowchart of a method for automatically extracting entity relationships in clinical trial qualification standards based on multi-round question-answering provided by an embodiment of the present invention;

[0042] Figure 2 is a schematic diagram of preprocessing text data provided by an embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of entity categories and relationship categories defined in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of a simple example of multi-round question-answering for automatic extraction of clinical trial qualification criteria based on the BERT-BiLSTM-CRF model provided in an embodiment of the present invention;

[0045] Figure 5It is a structural diagram of a system for automatically extracting entity relationships in clinical trial qualification standards based on multi-round question-answering provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0047] The inventive concept of the present invention is: in view of the low recall rate of the model extraction results in the current method for automatically extracting entity relations in the clinical trial qualification standards and the problem of error accumulation in the step-by-step extraction, the embodiment of the present invention provides a method and system for automatically extracting entity relations in the clinical trial qualification standards based on multi-round question-answering, adopts a multi-round question-answering paradigm to construct a method and system for automatically extracting entity relations in the clinical trial qualification standards, selects a BERT-BiLSTM-CRF model of deep learning and pre-trained language model, performs text preprocessing on the crawled original text of the clinical trial qualification standards, forms entity question positive samples, and constructs corresponding entity question negative samples, which together constitute a training data set, and trains the BERT-BiLSTM-CRF model to be a model suitable for automatic extraction of entity relations in the clinical trial qualification standards. In the prediction stage, by splicing the free text with the head entity question template and inputting it into the model, the head entity name is predicted, and then the name is replaced in the tail entity question template, spliced ​​with the free text and input into the model again to predict the tail entity name. Through the multi-round iterative question-answering process, the comprehensive extraction of entity relations in the clinical trial qualification standards is achieved.

[0048] Figure 1 FIG. 1 is a flow chart of a method for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question-answering provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a method for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question-answering, comprising the following steps:

[0049] S1, preprocessing including sentence segmentation and cleaning is performed on the acquired clinical trial eligibility criteria text to obtain the free text of the clinical trial eligibility criteria.

[0050] S1.1, Data acquisition. On the website of the Drug Clinical Trial Registration and Information Disclosure Platform, we searched for “tumor” as the keyword, and used the Selenium tool to automatically capture the eligibility criteria paragraphs in the fixed-format clinical trial web pages. The obtained data was stored as a text file for subsequent reading and preprocessing.

[0051] S1.2, data preprocessing. The collected original text of clinical trial eligibility criteria is segmented into sentences, using periods and semicolons as segmentation criteria to form single sentences of moderate length that contain complete semantics. The segmented text is cleaned to remove extra spaces and special symbols, and the character format is unified into Chinese to ensure the cleanliness and format consistency of the text. Figure 2 As shown, the left figure is the original text of the clinical trial eligibility criteria obtained, and the right figure is the free text obtained after preprocessing.

[0052] S2, based on the free text, define the entity categories and relationship categories in the free text, annotate the free text according to the defined entity categories and relationship categories to obtain the annotated data, and design the head entity question template and the tail entity question template for the head entity category and the tail entity category in the entity category respectively.

[0053] S2.1, entity relationship category definition. Refer to the "Common data elements for general and precision oncology clinical trials" table (source document: Zeng J, Shufean MA, Khotskaya Y, et al (2019) OCTANE: oncology clinicaltrial annotation engine. JCO clinical cancer informatics, 3, 1-11. https: / / doi.org / 10.1200 / CCI.18.00145), combined with the content in the free text corresponding to the actual crawled clinical trial eligibility criteria, conduct an in-depth analysis of the eligibility criteria text, and refine the definition of entity categories to ensure that each category has clear boundaries and descriptions. In the embodiment, 11 entity categories and 5 relationship categories are defined, such as Figure 3As shown in the figure, it aims to comprehensively cover various descriptions in the eligibility criteria. Among them, the 11 entity categories are divided into 8 head entity categories and 9 tail entity categories. The 8 head entity categories include: cancer type, treatment, drug, biopsy, gene type, stage, examination and variant type, and the 9 tail entity categories include: treatment, drug, biopsy, gene type, stage, time, result, variant type and value. Among them, the five relationship categories include conditional relationship (has_condition), time relationship (has_time), result relationship (has_result), variation relationship (has_change) and value relationship (has_value). The conditional relationship is used to represent the relationship between the head entity category (cancer type) and the tail entity category (treatment, drug, biopsy, genotype and stage). The time relationship is used to represent the relationship between the head entity category (treatment and drug) and the tail entity category (time). The result relationship is used to represent the relationship between the head entity category (biopsy) and the tail entity category (result). The variation relationship is used to represent the relationship between the head entity category (genotype) and the tail entity category (variation type). The value relationship is used to represent the relationship between the head entity category (examination) and the tail entity category (value).

[0054] S2.2, Data Annotation. 400 free texts of clinical trial eligibility criteria were randomly selected. Before entity annotation, an initial annotation guide was developed and continuously improved during the annotation process to ensure annotation consistency. The brat tool was used to manually annotate the entity and relationship names in the free text according to the predefined entity and relationship categories and annotation guidelines, and the .ann annotation file corresponding to each text was exported. Subsequently, a Python script was written to read the entity and relationship annotation data in each .ann file.

[0055] S2.3, question template design. Simulate the way humans ask questions, design a head entity question template based on the head entity category, and design a tail entity question template that lacks a subject based on the tail entity category. When asking questions, add the answer to the head entity question as the subject to the tail entity question template to form a tail entity question. The head entity question template is designed to guide the model to identify key entities, while the tail entity question template takes into account its relevance to the previous round of questions and answers, so the subject is omitted in the design to include the answer to the previous round, thereby forming a coherent question and answer process. By designing question templates for multiple rounds of questions and answers, the model can better understand the extraction task. Table 1 shows examples of head entity question and tail entity question templates.

[0056] Table 1 Examples of templates for head entity questions and tail entity questions

[0057]

[0058] In the embodiment, as shown in Table 1, "check" and "value" form a type of relationship, where "check" is the head entity and "value" is the tail entity. Each type of relationship has a corresponding question template, and "Head3" in the question template of "value" refers to the name of the head entity extracted by the model. The tail entity question template integrates the predefined relationship categories and can extract relationships and entities at the same time to a certain extent.

[0059] S3, constructs positive samples of head entity question and answer and tail entity question and answer based on free text, annotated data and question templates, constructs negative samples based on free text and unannotated entity categories, and inputs positive and negative samples into the BERT-BiLSTM-CRF model for model training.

[0060] The positive samples and negative samples are used as training sample sets, which contain question samples in the form of text and corresponding true label samples in the form of digital sequences.

[0061] Before constructing the real label sample in the form of a digital sequence, the BIO annotation format is used to store the entity boundaries and types. In the embodiment, 11 types of entity categories are defined, and each type has two annotation results, "B-" and "I-", and an "O" annotation, forming a total of 23 possible single entity word annotation results. In order to facilitate model processing, integer values ​​are assigned to these 23 category characters, where the corresponding number of "O" is 0, and the integer values ​​of other categories are continuously increased. When constructing the label, according to the known entity boundary index value, a digital sequence of all 0s is constructed, and the 0s therein are replaced with the corresponding entity numbers, thereby forming a real label in the form of a digital sequence for model training.

[0062] When constructing a text-based question sample, the free text and the entity question corresponding to the entity category are concatenated in the format of "[CLS,Q,SEP,C,SEP]". If it is a tail entity question, the corresponding head entity name is replaced in the tail entity question template.

[0063] S3.1, construct positive samples. In constructing positive samples, for individual entities or entity pairs obtained by manual annotation, the head entity question input and its corresponding true label, as well as the tail entity question input and its corresponding true label are constructed respectively.

[0064] (1) For individual entities present in free text:

[0065] The free text and the head entity question corresponding to the entity category are concatenated in the format of "[CLS,Q,SEP,C,SEP]" to generate a sample of positive head entity questions in text form. Then, a numeric sequence of all zeros is constructed as the answer label. For each category entity, if its entity question has only one answer in the free text, the 0 value of the answer position is replaced with the corresponding numeric number. If there are multiple answers, the 0 values ​​of all answer positions are replaced with the corresponding numeric numbers to obtain the true label of the positive answer.

[0066] (2) For entity pairs that constitute a relationship in free text:

[0067] When constructing the head entity question, the free text and the head entity question corresponding to the head entity category in the entity relationship are concatenated in the format of "[CLS,Q,SEP,C,SEP]" to generate a text-based positive head entity question sample. Similarly, a sequence of all zero numbers is constructed as the answer label. For each head entity question, if there is only one answer in the free text, the 0 value of the answer position is replaced with the corresponding number. If there are multiple answers, the 0 values ​​of all answer positions are replaced with the corresponding number to obtain the true label of the positive answer to the head entity question.

[0068] When constructing the tail entity question, if the head entity question has multiple answers in the free text, replace the head entity name into the tail entity question template corresponding to the tail entity category in the entity relationship to form multiple tail entity questions, which are concatenated with the free text in the format of "[CLS, Q, SEP, C, SEP]" to obtain positive tail entity question samples in text form. Construct a sequence of all zero numbers as the answer label. For each tail entity question, if it has only one answer in the free text, replace the 0 value of the answer position with the corresponding number. If there are multiple answers, replace the 0 values ​​of all answer positions with the corresponding number to obtain the true label of the positive answer to the tail entity question.

[0069] S3.2, construct negative samples.

[0070] (1) When constructing negative head entity samples, select head entity categories that do not appear in free text and are therefore not labeled, and concatenate the corresponding question templates with the free text in the format of "[CLS, Q, SEP, C, SEP]" to generate negative head entity question samples in text form. At the same time, since the head entities in these samples do not exist in the text, they are assigned empty labels, and all-zero digital sequences are constructed as negative samples of the corresponding answer labels to simulate entity categories that do not exist in the text, so as to train the model to learn to ignore entities that are irrelevant to the text.

[0071] (2) When constructing negative tail entity samples, for known head entity names, select tail entity categories that do not appear in the free text and are therefore not labeled, use question templates of these categories, replace the labeled head entity names into the tail entity question template to form a tail entity question, and concatenate the tail entity question with the free text in the format of "[CLS,Q,SEP,C,SEP]" to generate negative tail entity question samples in text form. Similarly, since these tail entities do not exist in the text, these samples are also assigned empty labels, and construct negative samples with all-zero digital sequences as corresponding answer labels to simulate entity relationships that do not exist in the text, so as to help the model learn to identify and exclude false entity relationships.

[0072] The number of negative sample sets constructed according to the above method is large. Some sample data are randomly selected, and the number of negative samples is continuously adjusted according to the training results to optimize the generalization ability of the model and the efficiency of training.

[0073] S3.3, model training. The BERT-BiLSTM-CRF model constructed in the embodiment is a model combining BERT, BiLSTM (bidirectional long short-term memory network) and CRF (conditional random field), wherein a full-word masked Chinese BERT model is adopted, which is a pre-trained language model optimized for Chinese scenes, wherein the full-word masking strategy is particularly critical, and the entire complete Chinese vocabulary is masked as a unit during the pre-training process, so as to more accurately retain and learn the integrity and context of Chinese vocabulary. Based on the statistics of the length of the free text after pretreatment, the longest input text length that BERT can process is set to 150 characters. Regardless of the head and tail questions and answers, the input format of the positive sample and the negative sample adopts "[CLS, Q, SEP, C, SEP]", wherein "C" represents free text, "Q" represents entity questions, "CLS" represents input sequence tags, and "SEP" represents separator tags. After the sample is processed by the BERT word segmenter, the output vector is processed by the high-dimensional processing of the BERT-BiLSTM-CRF model, and the most likely label sequence is predicted.

[0074] S4: In each round of question-answering, the head entity question is concatenated with the free text and then input into the trained BERT-BiLSTM-CRF model to predict the head entity name. The head entity name is then replaced into the corresponding tail entity question and concatenated with the free text again and then input into the model to predict the tail entity name. Through multiple rounds of question-answering, the model can finally predict all entities and their relationships in the free text.

[0075] like Figure 4 A simple example in the embodiment is shown, taking the eligibility criteria sentence about cancer types "unresectable advanced hepatocellular carcinoma" in free text as an example, demonstrating the question-answering process of entity and relationship extraction.

[0076] S4.1, head entity extraction. Figure 4 As shown in the figure, the model focuses on identifying entities in the text that do not have inherent relations or are head entities in relations. The head entity questions and free text are concatenated in the format of "[CLS,Q,SEP,C,SEP]" and input into the BERT-BiLSTM-CRF model. These questions are input into the BERT tokenizer and encoded into three vectors: tokenembeddings, segmentembeddings, and positionembeddings. The vectors encoded for all head entity questions are superimposed together and input into the BERT-BiLSTM-CRF model for further high-dimensional processing. The most likely prediction sequence is output in the format of a digital sequence. According to the mapping relationship between digital codes and category characters established in the model training phase, the digital sequence is converted into a character sequence in BIO format. The continuous character string of the non-"O" part of the sequence is extracted and converted into the head entity name of the corresponding position in the free text, thereby obtaining the entity name answer of all head entity questions.

[0077] S4.2, tail entity extraction. Figure 4 As shown in the figure, the model aims to extract tail entities that can establish a relationship with the previously identified head entity. According to the head entity category corresponding to the head entity name, the head entity name is replaced into the "Head" part of all corresponding tail entity question templates, and is concatenated with the free text in the "[CLS,Q,SEP,C,SEP]" format to generate one or more tail entity questions, which are also superimposed and input into the model for prediction. The predicted digital sequence is converted into a character sequence, and the continuous string of non-"O" parts in the character sequence corresponds to the entity name answer of the tail entity question in the free text.

[0078] In multiple rounds of question answering, the model recognizes all head entity names in the free text by iterating different head entity questions. Each head entity name is replaced with the subject part in the tail entity question, and the model recognizes all tail entity names and corresponding relationship categories that form a relationship with the head entity name in the free text by iterating different tail entity questions. If there is no entity category corresponding to the question in the free text, the model will output None, indicating that there is no relevant entity.

[0079] In summary, a method for automatically extracting entity relationships in clinical trial eligibility criteria based on multi-round question-answering, using the BERT-BiLSTM-CRF model, is very effective in information extraction. Thanks to BiLSTM's ability to capture word order interactions, the model can effectively integrate contextual information, thereby strengthening the understanding of semantics. The BERT model's advantages in grasping sentence-level semantics and long-distance dependencies further improve the accuracy of extraction. At the same time, the multi-round question-answering paradigm is adopted to demonstrate effective semantic understanding by converting entity extraction into a human question-answering task. This method provides prior information on the target entity category, merges entity and relationship extraction into a unified task, and helps obtain accurate results. By combining the multi-round question-answering paradigm with the excellent BERT-BiLSTM-CRF model, the model's semantic understanding ability can be further enhanced and act as a joint extraction framework, reducing error accumulation and improving recall.

[0080] Based on the same inventive concept, Figure 5 As shown, an embodiment of the present invention also provides a system 500 for automatically extracting entity relationships in clinical trial qualification standards based on multi-round questions and answers, including: a data acquisition and preprocessing module 510, a model training preparation module 520, a model training module 530 and an entity relationship extraction prediction module 540.

[0081] The data acquisition and preprocessing module 510 is used to perform preprocessing including sentence segmentation and cleaning on the acquired clinical trial qualification standard text to obtain a free text of the clinical trial qualification standard.

[0082] The model training preparation module 520 is used to define entity categories and relationship categories based on free text, annotate the free text according to the defined entity categories and relationship categories to obtain annotated data, and design a head entity question template and a tail entity question template for the head entity category and the tail entity category in the entity category respectively.

[0083] The model training module 530 is used to construct positive samples of head entity question and answer and tail entity question and answer based on free text, annotated data and question templates, construct negative samples based on free text and unannotated entity categories, and input the positive samples and negative samples into the BERT-BiLSTM-CRF model for model training.

[0084] The entity relationship extraction prediction module 540 is used to concatenate the head entity question with the free text in each round of question and answer, and then input the concatenated data into the trained BERT-BiLSTM-CRF model to predict the head entity name, and then replace the head entity name with the corresponding tail entity question and concatenate it with the free text again and input the concatenated data into the model to predict the tail entity name. Through multiple rounds of question and answer, the model can finally predict all entities and their relationships in the free text.

[0085] Based on the same inventive concept, an embodiment of the present invention also provides a device for automatically extracting entity relationships in clinical trial qualification standards based on multiple rounds of questions and answers, including a memory and one or more processors, the memory is used to store computer programs, and the processor is used to implement the above-mentioned method for automatically extracting entity relationships in clinical trial qualification standards based on multiple rounds of questions and answers when executing the computer program.

[0086] Based on the same inventive concept, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned method for automatically extracting entity relationships in the clinical trial qualification criteria based on multiple rounds of questions and answers is implemented.

[0087] It should be noted that the above-mentioned embodiments provide a system for automatically extracting entity relationships in clinical trial qualification standards based on multiple rounds of questions and answers, a device for automatically extracting entity relationships in clinical trial qualification standards based on multiple rounds of questions and answers, and a computer-readable storage medium, all of which belong to the same inventive concept as a method for automatically extracting entity relationships in clinical trial qualification standards based on multiple rounds of questions and answers. The specific implementation process is detailed in an embodiment of a method for automatically extracting entity relationships in clinical trial qualification standards based on multiple rounds of questions and answers, which will not be repeated here.

[0088] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatically extracting entity relationships in clinical trial eligibility criteria based on multi-round question-answering, characterized in that: The following steps are involved: Perform preprocessing including sentence segmentation and cleaning on the acquired clinical trial eligibility criteria text to obtain the free text of the clinical trial eligibility criteria; Based on the free text, the entity categories and relationship categories are defined, the free text is annotated according to the defined entity categories and relationship categories to obtain annotated data, and a head entity question template and a tail entity question template are designed for the head entity category and the tail entity category in the entity category respectively; Based on free text, annotated data, and question templates, construct positive samples of head entity question answering and tail entity question answering. Based on free text and unannotated entity categories, construct negative samples. Input positive and negative samples into the BERT-BiLSTM-CRF model for model training. In each round of question-answering, the head entity question is concatenated with the free text and input into the trained BERT-BiLSTM-CRF model to predict the head entity name. The head entity name is then replaced into the corresponding tail entity question and concatenated with the free text again and input into the model to predict the tail entity name. Through multiple rounds of question-answering, the model can eventually predict all entities and their relationships in the free text.

2. The method for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question-answering according to claim 1, characterized in that: The entity categories and relationship categories defined based on free text include: Entity words or phrases in free text are analyzed and a head entity category and a tail entity category, as well as a relationship category between the head entity category and the tail entity category, are defined, wherein the head entity category is the entity category as the starting point of the relationship, and the tail entity category is the entity category as the end point of the relationship.

3. The method for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question-answering according to claim 1, characterized in that: The step of annotating the free text according to the defined entity categories and relationship categories to obtain annotated text data includes: Using the brat tool, the entities in the free text are annotated with entity categories and relationship categories according to the entity categories and relationship categories and the established annotation guidelines, and the annotated text data corresponding to each free text is obtained.

4. The method for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question-answering according to claim 1, characterized in that: The step of designing a head entity question template and a tail entity question template for the head entity category and the tail entity category in the entity category respectively includes: Simulating the way humans ask questions, a head entity question template is designed according to the head entity category, and a tail entity question template that lacks a subject is designed according to the tail entity category. When asking and answering questions, the answer to the head entity question is added as the subject to the tail entity question template to form a tail entity question.

5. The method for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question-answering according to claim 1, characterized in that: The positive samples of constructing head entity question and answer and tail entity question and answer based on free text, annotated data and question template include: For a single head entity category, the corresponding head entity question is concatenated with the free text to construct a positive head entity question input and a positive answer true label; For the head and tail entity category pairs, the head entity question corresponding to the head entity category is concatenated with the free text to form a positive head entity question input and the corresponding positive answer true label is given. The answer to the head entity question is added to the tail entity question template corresponding to the tail entity category to form a tail entity question. The tail entity question is concatenated with the free text to form a positive tail entity question input and the corresponding positive answer true label is given.

6. The method for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question-answering according to claim 1, characterized in that: The negative samples are constructed based on free text and unlabeled entity categories, including: Select the head entity categories that do not appear in the free text and are therefore not labeled, concatenate the corresponding head entity questions with the free text to construct a negative head entity question input and give the corresponding negative answer empty label; Select the tail entity category that does not appear in the free text and is therefore not labeled, replace the labeled head entity name into the tail entity question template to form a tail entity question, concatenate the tail entity question with the free text to construct a negative tail entity question input and give the corresponding negative answer empty label.

7. The method for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question-answering according to claim 1, characterized in that: Through multiple rounds of question-answering, the model finally predicts all entities and their relationships in the free text, including: In multiple rounds of question answering, the model recognizes all head entity names in free text by iterating different head entity questions; for each head entity name, it is replaced with the subject part in the tail entity question, and the model recognizes all tail entity names and corresponding relationship categories that have a relationship with the head entity name in the free text by iterating different tail entity questions; if the entity category corresponding to the question does not exist in the free text, the model will output no relevant entity.

8. A system for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question-answering, characterized in that: include: Data acquisition and preprocessing module, model training preparation module, model training module and entity relationship extraction and prediction module; The data acquisition and preprocessing module is used to perform preprocessing including sentence segmentation and cleaning on the acquired clinical trial qualification standard text to obtain a free text of the clinical trial qualification standard; The model training preparation module is used to define entity categories and relationship categories in the free text based on the free text, annotate the free text according to the defined entity categories and relationship categories to obtain annotated data, and design a head entity question template and a tail entity question template for the head entity category and the tail entity category in the entity category respectively; The model training module is used to construct positive samples of head entity question and answer and tail entity question and answer based on free text, annotated data and question templates, construct negative samples based on free text and unannotated entity categories, and input the positive samples and negative samples into the BERT-BiLSTM-CRF model for model training; The entity relationship extraction prediction module is used to splice the head entity question with the free text in each round of question and answer, and then input the spliced ​​head entity question into the trained BERT-BiLSTM-CRF model to predict the head entity name, and then replace the head entity name into the corresponding tail entity question and splice it with the free text again and input the model to predict the tail entity name. Through multiple rounds of question and answer, the model finally predicts all entities and their relationships in the free text.

9. A device for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question-answering, comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the method for automatically extracting entity relationships in clinical trial qualification criteria based on multi-round question and answer as described in any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer, the method for automatically extracting entity relationships in clinical trial qualification criteria based on multiple rounds of questions and answers as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Training method and device of medical named entity recognition model, equipment and medium

    CN113724819A

  • Generative information extraction method and device based on pre-training model

    CN113780011A

  • Entity relation joint extraction method based on automatic question and answer

    CN113836281A

  • CT image report information extraction method and device based on multi-round questions and answers, computer equipment and storage medium

    CN114360677A

  • Context tag integration with named entity recognition models

    US20220229993A1